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Abstract 

Direct manipulation interfaces and programmatic systems 
have distinct and complementary strengths. The former pro¬ 
vide intuitive, immediate visual feedback and enable rapid 
prototyping, whereas the latter enable complex, reusable ab¬ 
stractions. Unfortunately, existing systems typically force 
users into just one of these two interaction modes. 

We present a system called Sketch-n-Sketch that 
integrates programmatic and direct manipulation for the 
particular domain of Scalable Vector Graphics (SVG). In 
Sketch-n-Sketch, the user writes a program to generate 
an output SVG canvas. Then the user may directly manip¬ 
ulate the canvas while the system immediately infers a pro¬ 
gram update in order to match the changes to the output, a 
workflow we call live synchronization. To achieve this, we 
propose (i) a technique called trace-based program synthe¬ 
sis that takes program execution history into account in order 
to constrain the search space and (ii) heuristics for dealing 
with ambiguities. Based on our experience with examples 
spanning 2,000 lines of code and from the results of a pre¬ 
liminary user study, we believe that Sketch-n-Sketch 
provides a novel workflow that can augment traditional pro¬ 
gramming systems. Our approach may serve as the basis for 
live synchronization in other application domains, as well as 
a starting point for yet more ambitious ways of combining 
programmatic and direct manipulation. 

Categories and Subject Descriptors D.3.3 [Programming 
Languages] : Language Constructs and Features; F.3.2 [Log¬ 
ics and Meanings of Programs]: Program Analysis; H.5.2 
[Information Systems Applications]: User Interfaces 

Keywords Sketch-n-Sketch, Prodirect Manipulation, SVG 
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1. Introduction 

Direct manipulation user interfaces ll29l such as Adobe Il¬ 
lustrator, Microsoft PowerPoint, and GIMP ini and pro¬ 
grammatic systems such as Processing Enm each have 
distinct strengths. The former provide intuitive, immediate 
visual feedback and enable rapid prototyping, whereas the 
latter allow for designing and reusing complex abstractions. 
Neither mode of interaction is preferable for all tasks. 

Motivation. Imagine designing a series 
of identical shapes, laid out along the con¬ 
tours of a sine wave. Designing the first 
“prototype” shape using a visual, direct 
manipulation tool like Illustrator or Pow¬ 
erPoint works well. But after copying the shape, pasting 
it multiple times becomes tedious, and achieving the sinu¬ 
soidal pattern requires painstaking effort because built-in 
features (e.g. rulers, snapping to other shapes, and uniform 
spacing) are unlikely to help. Worse yet are the edits required 
to change high-level parameters of this design (e.g. the num¬ 
ber of shapes, the spacing between them, or the amplitude 
of the sine wave). The user will want to scrap the previous 
effort and start from scratch. On the other hand, by using a 
programmatic system, it is easy to churn out variations of the 
high-level pattern by changing parameters and re-running 
the program. But the disconnect between the software ar¬ 
tifact and its output can limit the pace of design, especially 
when the appropriate parameters in the program are difficult 
to identify. 

In a recent position paper, we identified several domains 
for both visual and textual data where users are forced to 
make the unfortunate choice between programmatic or direct 
manipulation tools 1121 We proposed the notion that soft¬ 
ware systems ought to smoothly integrate these two modes 
of use, a combination we call prodirect manipulation. Ide¬ 
ally, a user could manipulate the output of a program and the 
system would simultaneously update the program to keep it 
synchronized with the changes. In this paper, we present the 
first realization of that goal. 

Challenges. Suppose that a program e generates k output 
values and that a user action changes the first j values: 
{vi v[, ..., Vj Vp T/j+i, ..., Vk}. To synthesize a 
program e' that generates the updated output, three primary 


challenges must be addressed: (1) The meaning of the user’s 
updates must be defined in a way that constrains the program 
synthesis search space; (2) The search algorithm must run 
quickly and find program updates that are easy for the user to 
understand; and (3) When multiple candidate solutions exist, 
the ambiguity must be dealt with in a way that facilitates 
the responsive, interactive workflow characteristic of direct 
manipulation interfaces. 

Key Idea 1: Trace-Based Program Synthesis. To address 
challenge (1), we propose a technique called trace-based 
program synthesis that comprises two components. First, we 
instrument the evaluation of the program e so that each of the 
k values Vi it produces comes with a trace ti, which forms a 
set of constraints {vi = fi, ..., Vk = tk} relating the pro¬ 
gram to its output. Second, to reconcile e with the j updated 
values, we specify that an ideal candidate update would be a 
program e' whose output satisfies the system of constraints 
C - - ^1: • • • 7 + l • • • 5 ^/c - ^/c}* 

Key Idea 2: Small Updates. To address challenge (2), our 
design principle is that only “small” program updates may 
be inferred; “large” changes may require user intervention 
and are thus less conducive to our goal of immediate syn¬ 
chronization. In particular, we attempt only to change value 
literals in the program, represented as a substitution p that 
maps locations in the abstract syntax tree to new values. 

Key Idea 3: Heuristics for Disambiguation. Even with 
the seemingly modest goal of inferring small updates, there 
is still the problem of dealing with multiple solutions. To 
address challenge (3), we first decompose C into k separate 
constraints Ci = {v[ = ti} through Ck = {v'f^ = tk} and 
then use heuristics to force each Ci to have at most one 
solution pi. This enables us to define a trigger function 
X{v[, ..., Vj). p that, given the concrete values changed by 
the user, computes a solution pi for each constraint Ci and 
combines them into a single substitution p = pi • • • p/c. This 
substitution is applied to the original program in real-time 
during the user action, resulting in a new program pe that is 
evaluated and ready for subsequent user manipulation. 

Contributions and Outline. These three key ideas are gen¬ 
eral and may be developed for several settings. In this paper, 
we focus on instantiating our approach for the specific do¬ 
main of Scalable Vector Graphics (SVG). In particular: 

• We propose a framework for inferring program updates 
called trace-based synthesis, which may apply broadly 
to domains where users wish to (indirectly) manipulate 
programs by (directly) manipulating their outputs. ( [§^ 

• We define heuristics for disambiguating between possible 
updates and define triggers that keep a program synchro¬ 
nized with changes to its SVG output in real-time. ( [§4| ) 

• We implement our ideas in a Web-based system called 
Sketch-n-Sketch and evaluate its interactivity, ( [p] ) 


• We use Sketch-n-Sketch to design many examples 
difficult to develop or edit using existing tools. 

Our implementation and examples, as well as additional 
tutorial documentation and videos, are publicly available at 
http://ravichugh.github.io/sketch-n-sketch 


2. Overview 


We will provide an overview of Sketch-n-Sketch by 
considering the program in [Figure T] that generates the sine 
wave box design in §J_ Suppose the user clicks on the third 
box and drags it to a new position down and to the right (de¬ 
picted in [Figure T] 2). In this section, we will describe how 
Sketch-n-Sketch synthesizes four candidate updates to 
the program, each of which computes the position of the 
third box to match the user’s direct manipulation but have 


different effects on the remaining boxes (depicted in Fig- 
|ure Ip ). 


A Little Programming Language. The programming lan¬ 
guage in Sketch-n-Sketch is a core, untyped func¬ 
tional language called little, which includes base val¬ 
ues (floating-point numbers, booleans, and strings) and lists 
(represented as cons-cells, or pairs). For reference, we define 
the syntax of little in Figure 2[ All expression and value 
forms are standard except for numbers. 

Because numeric attributes are often directly manipulated 
in graphical user interfaces, each numeric literal in little 
comes with three pieces of additional information: a location 
i, which is an integer inserted by the parser that identifies its 
position in the abstract syntax tree (AST); an optional, user- 
specified location annotation a, where ! is used to freeze the 
number; and an optional, user-specified range annotation /3, 
where { ni - n 2 } denotes the intended range of the number. 
We discuss these annotations further later in this section. 


Representing SVG Values. An SVG node is represented 
as a list [ svgNodeKind attributes children ] with 
three values where the string svgNodeKind describes the 
kind of the node (e.g. ^rect\ ^ circle \ or ^line^ for 
particular shapes, or ^ svg ^ for a canvas, or collection, of 
shapes); attributes is a list of attributes (i.e. key-value 
pairs); and children is a list of child nodes. The intended 
result of a little program is a node with kind ^ svg\ The 
result of evaluating the program in [Figure takes the form 
below, which gets translated directly to the SVG format Ea 
and resembles the screenshot in when rendered. 


['svg' [] 

[ ['reef [['x' 50] ['y' 120] ...] [] ] 

['reef [['x' 80] ['y' 90] ...] [] ] 

['reef [['x' 110] ['y' 68] ...] [] ] ... ] ] 


Our translation of key-value attributes provides a thin wrap¬ 
per over the target SVG format, allowing little programs 
to specify arbitrary XML attributes using strings. We discuss 
the translation further in Supplementary Appendices ca, 
but the details are not needed in the rest of the paper. 

















(A) Excerpt from prelude. little 

(defrec range (A(i j) 

(if (> i j) nil 

(cons i (range (+ i) j))))) 

(def zeroTo (An (range (- n 1)))) 

(B) sineWaveOfBoxes.little 

(def [xO yO w h sep amp] [50 120 20 90 30 60]) 
(def n 12!{3-30}) 

(def boxi (Ai 

(let xi (+ xO (* i sep)) 

(let yi (- yO (* amp (sin (* i (/ twoPi n))))) 
(rect ^lightblue^ xi yi w h))))) 

(svg (map boxi (zeroTo n))) 

(C) Suppose the user clicks on the third box from the left 
(colored darker in red for emphasis) and drags it to a new 
position down and to the right (colored lighter in gray): 


I 


(D) Sketch-n-Sketch synthesizes four candidate 
updates to the program, which have the following effects: 



Figure 1. Sine Wave of Boxes in Sketch-n-Sketch 


2.1 Locations and Traces 

To enable direct manipulation, the little evaluator pro¬ 
duces run-time traces to track the evaluation of numeric 
values (for attributes such as ^y\ ^width\ and 

^height 0- There are two kinds of traces t that are used to 
infer program updates based on direct manipulation changes. 


Locations. The simplest kind of trace records that a nu¬ 
meric literal n originates from a particular source-code lo- 
cation I in the AST of the program. For the program in 


Figure 1 3, the little parser annotates all numbers 50 


^2 


120^3, 20^^, 90^^ 30^®, 60^"^, and 12^® with unique location 


identifiers. These locations do not affect program evaluation 
or SVG rendering. When the user manipulates a numeric 
value in the canvas {e.g. by dragging or stretching a box), 
Sketch-n-Sketch updates the value at program location 
I in real-time. 

In the rest of the paper, when a number n is immediately 
bound to a variable x, we choose the canonical name x for 
the location — resulting in the value — rather than one of 
the form Ik • In the examples that follow, we sometimes an¬ 
notate numeric literals explicitly with locations for explana¬ 
tory purposes even though the programmer does not write 
them — they are inserted implicitly by our parser. 

Expression Traces. Smaller traces are combined into 
larger ones during the evaluation of primitive operations, 
such as addition, multiplication, etc. Whereas the ^ width ^ 
and ^height ^ attribute values originate from atomic AST 
locations, the ^ x ^ value of each box is the result of evalu¬ 
ating xi = (+ xO (* i sep)) with different bindings for 
i. Each run-time value that is bound to i is generated by the 
function zeroTo (from a Prelude library included in every 
program), which computes the list of integers from 0 to n-1. 

When evaluating a primitive operation, the run-time se¬ 
mantics of little records the structure of the expression 
along with the resulting value. Below are the values of xi 
for each of the first three boxes, respectively, and their cor¬ 
responding traces. Each value-trace pair forms an equation: 

50 = (+ xo (* 4 sep)) ( 1 ) 

80 = (+ Xo (* (+ h io) sep) ) ( 2 ) 

110 = (+ Xo (* (+ ii (+ io)) sep) ) ( 3 ) 

The locations io and ii identify literals 0 and 1, respectively, 

in the Prelude function zeroTo that computes increasing 
integers. The value-trace equations for the remaining boxes 
are analogous. These equations, together with the following 
substitution that records location-value mappings from the 
source program, relate the program to its output. 

Po — \_xo I —y 50 , sep I —y 30 , io i^ 0 , i\ i —y 1 , • • •] 

Dataflow-Only Traces. The reduction rule E-Op-Num 
( [F igure 2) builds expression traces in parallel with the eval¬ 
uation of primitive operations, producing values n^. Traces 
t record data fiow but not control flow. This design is based 
on the insight that programs generating output in visual do¬ 
mains are often structured so that the control fiow of the 
program is similar across multiple runs. We have not found 
this limitation to be a problem in practice for the examples 
we have developed. However, it may be useful to enrich 
traces with additional information in subsequent work. 

2.2 Synthesizing Program Updates 

The main idea behind trace-based program synthesis is to 
use value-trace equations in order to infer program updates 












that conform to output values changed by the user. In the set¬ 
ting of direct manipulation interfaces, changing attributes of 
visual objects corresponds to changing the values on the left- 
hand side of the equations above. In our sineWaveOf Boxes 
example, the result of dragging the third box directly to the 


right so that its new value is 155, is to replace Equa¬ 


tion 3 with the following equation: 

155 = (+ xo (* (+ h (+ h 4) ) sep )) 


( 3 ’) 


Our goal is to synthesize an updated program that satis¬ 
fies Equation 3’; satisfying all other (unchanged) equations 
would be ideal, but this one is “more important” because it 
was induced by the user’s change. 


Local Updates. We aim only to infer local updates, which 
are substitutions that map locations to updated numeric val¬ 
ues. We describe design and implementation decisions in 
and |§ 5| that limit the cost of solving equations to en¬ 


sure responsive interaction with the user. With our approach, 
Sketch-n-Sketch can infer four substitutions based on 


Equation 3’ pi = po[xo 95], p 2 = po[sep ^ 52.5], 


Ps — ^ 1-5]? and p4 — po[£i ^ 1.75]. 


These substitutions, if applied to the program in Eig- 


ure Ip , would produce various effects. The first option 


would translate all of the boxes in unison. The second would 
increase the spacing between boxes. The third and fourth 
would, respectively, translate all boxes or the change the 
separation, but both would also change the number of boxes 
because the constants 0 and 1 at locations Iq and ii were 
used in the original program to compute integer indices. 
The user is unlikely to want either the third or fourth op¬ 
tions, since the list of integers 0 to n-1 specifies the number 
of boxes. Moreover, the locations io and ii appear in the 
Prelude, so changing their values would affect the behav¬ 
ior of other programs! 


Frozen Constants. By default, Sketch-n-Sketch will 
consider updating the value of any program location used 
in a value-trace equation in order to reconcile the program 
with the user’s changes. The user can direct the synthe¬ 
sis procedure not to change the value of particular con¬ 
stants by freezing them, denoted with exclamation points 
{e.g. 3.14!). All numbers in Prelude are automatically 
frozen, so solutions ps and p 4 are not actually considered 
by Sketch-n-Sketch. Without freezing either xO or sep, 
however, the ambiguity between pi and p 2 remains. 


2.3 Heuristics 

Pausing and asking the user to choose between updating xq 
or sep would stymie the interactive, live synchronization we 
strive for. Instead, we employ heuristics for automatically re¬ 
solving ambiguities that attempt to strike a balance between 
interactivity and predictability. Our key insight is that the 
essence of a local update is the set of constants C (i.e. pro¬ 
gram locations) that are changed and not necessarily their 


new values. Even if we arbitrarily decide that a particular 
user action should cause a set jCi of constants to change 
rather than a set £ 2 , we can often assign a different user 
action to manipulate the constants in £ 2 - 

If the user drags the first box horizontally to a new ^ x ^ 
position n, the change would induce the value-trace equa¬ 
tion n = (+Xo io sep)) based on Equation 1 which 
can be solved by changing the value of either xq or sep. 
In preparation, we arbitrarily choose to update xq to n in or¬ 
der to solve the new equation if and when the user drags 
this box. If the user drags the second box, again, either 
Xo or sep could be changed in order to solve the equa¬ 


tion n = (+Xo (* (+ii io) sep)) based on Equation 2 


Because we already assigned xo to vary if the user drags 
the first box, we choose to vary sep if the user drags the 
second box. If the user drags the third box, again, either 
Xo or sep could be changed to solve the induced equation 


n = (+ Xo (* (+^i (+^i 4)) sep)) based on [Equation 3 


Because each location has already been assigned to vary in 
response to some user action, we arbitrarily choose xo . 

We continue to assign program updates in this fashion 
by trying to balance the number of times a location set is 
assigned to some user action in the canvas. When the user 
performs an action, we use the pre-determined location set, 
together with the concrete values from the mouse manipu¬ 
lation, to compute a local update {i.e. substitution), apply it 
to the original program, run the resulting new program, and 
render the new output canvas. 


2.4 Sliders 

There are several situations in which it can be difficult or im¬ 
possible to make a desired change to the program via direct 
manipulation: (1) when there is no natural, visual representa¬ 
tion of the program parameter of interest; (2) when a visual 
representation may be hard to directly manipulate, for ex¬ 
ample, because the shape is too small or there are too many 
adjacent shapes; and (3) when ambiguous updates are hard 
to resolve using the built-in heuristics or by deciding which 
constants to freeze or not. 

Sketch-n-Sketch provides a feature that can help in 
all three situations. If a number is annotated with a range, 
written {nmin~ nmax}, then Sketch-n-Sketch will 
display a slider in the output pane that can be used to ma¬ 
nipulate the n value between rimin and Umax (as opposed to 
having to edit the program). Therefore, sliders can provide 
control over otherwise hard-to-manipulate attributes. 

In the sineWaveOf Boxes exam¬ 
ple, the number of boxes n is the ^ n: 12 ] 

only parameter that is hard to di¬ 
rectly manipulate. The heuristics choose to update n in re¬ 
sponse to certain actions, but the effects are not intuitive. 
Therefore, on line 2 of |Eigure~T^ , we freeze the value of n 
(so that it will never change as a result of directly manipulat¬ 
ing the boxes) and declare the range {3-30} so that we can 
easily adjust the number of boxes using a slider. 




















Syntax of Expressions 

e ::= N \ s \ b \ [] | [ei 1 62! | x 

I (Ape) I (6162) I (opmei’-em) 

I (let pei 62) I (letrec p ei 62) 

I (case e (pi ei) • • • (pm e^)) 

p ::= X \ n \ s \ b \ n \ [pi\p2l 

N ::= {n^,a,f 3 ) a ::= • \ \ /3 ::= • | {711-712} 

opo ^ {pi,.. •} 

opi G {not,cos,sin, arccos, round, floor,...} 
op2 ^ mod, pow,...} 

Syntax of Values 


of our workflow. Next, we will formally define what consti¬ 
tutes a valid solution to a system of constraints. 

Contexts and Substitutions, We define a value context V 
below to be a value with 777 > 0 placeholders, or holes, 
labeled •! through •^. We define the application of a 
value context to a list of values as • ^Vm) = 

• • '[xm/^m]• A substitution p is a mapping from 
program locations I to numbers n. When applied to an ex¬ 
pression, the bindings of a substitution are applied from left- 
to-right. Thus, the rightmost binding of any location takes 
precedence. We use juxtaposition p p' to denote concatena¬ 
tion, and we write p 0 1 -^ 77 ) to denote p 1 -^ 77 ]. We 

define value context similarity below to relate values that are 
structurally equal up to the values of numeric constants. 


v,w ::= 77^ I s I 6 I [] I [vi\v 2 l \ (Ape) 

t 11= ^ \ (oPm tl • • • tra) 


Operational Semantics (excerpted) 


e J} u 


n = [ (opm ^1 • • • rim) I t = (oPm h - ' tm) 

(opm ' ’ ’ Xljn ) {t 77 


[E-Op-Num] 


Figure 2. Syntax and Semantics of little 


3. Trace-Based Program Synthesis 

In the previous section, we defined the semantics of little 
to produce run-time traces for numeric attributes. In this sec¬ 
tion, we formulate trace-based program synthesis as a way 
to define the relationship between a program and updates to 
its output, without regard to the particular domain. 

User Actions. Suppose that a little program e generates 
output that contains k numeric values. With a single action, 
the user may directly manipulate j of the numeric values, for 
some l<j<k. The following table shows how, as alluded to 
in[§T] the j updated values, together with the k—j unchanged 
ones, form a system of constraints that, ideally, an updated 
program e' would satisfy: 


Program 

Output 

Updates 

Constraints 


II 

n'l 

n'l =ti (hard) 




(hard) 

e ^ 

Wj = np 

Wk = np 

^n' 

77' = tj (hard) 
(soft) 

rik = tk (soft) 


The user’s changes may lead to an unsatisfiable set of equa¬ 
tions (when considering only local updates). We treat equa¬ 
tions induced by changes as “hard” constraints that a solu¬ 
tion ought to satisfy, whereas the rest are “soft” constraints 
that should be satisfied if possible. This design principle pri¬ 
oritizes explicit changes made by the user, which is the goal 


V ::= •i \ \ s \ b \ [] | [Vill/z] | (Ape) 

Vi ~ V{ ^2 ~ 

Fi ni* - n2* [Vi I F 2 ] ~ Lv; I ^ 2 '] 

Definition: Faithful Updates. If 

(a) e 7;, where v = V (tci, ..., Wk)’, and 

(b) the user updates tci, ..., wj to ..., w'j, 

then a substitution p is faithful if 

(c) pe J} 7 ;' = ..., w'l) where V' ^V', implies 

(d) w'l = w[ for all 1 < 7 < j. 

Premises (a) and (b) identify the list of j values manipulated 
by the user, and properties (c) and (d) capture the notion that 
hard constraints induced by these changes should be satisfied 
by the update p. The value similarity relation checks that two 
value contexts are structurally equal but says nothing about 
the soft constraints from the original program (namely, it 
does not say w'l = Wi for all j < 7 < /c). In a setting where 
multiple updates are synthesized, ranking functions could be 
used to optimize for soft constraints. 

It is important to note that our definition states “(c) im¬ 
plies (d)” rather than the stronger property “(c) and (d)” be¬ 
cause the control fiow may change and produce V' V. 
We choose the weaker version because we do not intend to 
reason about control fiow either in traces or our synthesis al¬ 
gorithm ( |§5| ) when considering how one program compares 
to another. In other settings, it may be worthwhile to require 
the stronger version, which would necessitate a richer trace 
language that records control-flow information. 

Definition: Plausible Updates. We define an alternative, 
weaker correctness criterion. In particular, we define a plau¬ 
sible update to be one that satisfies some (i.e. at least one) of 
the user’s updates. Concretely, a plausible update is defined 
just like a faithful one, except that the following condition 
replaces (d) in the original definition: 

(d’) 77 ;'' = w'- for some I < i < j 












The general framework presented in this section can be 
instantiated with solvers (which we will refer to as Solve) 
that aim for different points along this spectrum of faithful 
and plausible updates. 

4. Live Synchronization for SVG 

Given changes to the output of a program, in the previous 
section we defined how value-trace equations can be used to 
specify candidate program updates in order to reconcile the 
changes. In this section, we describe how to compute pro¬ 
gram updates in real-time for the specific domain of Scal¬ 
able Vector Graphics (SVG). First, we identify what con¬ 
stitutes a user action in this setting. Second, we formulate 
how to compute triggers that dictate program updates based 
on such actions. For the latter, we propose heuristics to au¬ 
tomatically resolve ambiguities that result from trace-based 
program synthesis problem instances. 

In the following, we write ^ ^ k ^ ] to refer to the value of 
attribute ^ k Mn the little SVG value v. We also define the 
abbreviations Num(n^) = n and Tr(n^) = t. 

User Actions, Consider a value r that represents a rectan¬ 
gle positioned at (r[^x^],r[^y^]) = {rix^^ ,ny^y). Suppose 
the user clicks the mouse button somewhere inside the bor¬ 
ders of r (rendered visually) and then drags the cursor dx 
pixels in the x-direction and dy pixels in the ^-direction. As 
a result, the new desired position of r is given by (n^, n'y) = 
{ux dx^Uy ^ dy). Our goal is to reconcile this change to 
the position of r with the original program that generated 
it. One option is to wait until the user finishes dragging the 
rectangle, that is, when the user releases the mouse button. 
At that point, we could invoke Solve({n^ = tx^n'y = ty}) to 
compute a set of substitutions. Our goal with live synchro¬ 
nization, however, is to immediately a apply program update 
during the user’s actions. 

4.1 Mouse Triggers 

When the user clicks on a shape, we compute a mouse 
trigger r = X{dx^ dy). p, which is a function that, based 
on the distance the mouse has moved, returns a substitution 
to be immediately applied to the program. 

For now, let us assume that all shapes are rectangles 
and that user actions manipulate only their ^ x ^ and ^ y ^ at¬ 
tributes. For every shape in the canvas, there are two steps 
to compute a mouse trigger. First, for each attribute ^ x ^ and 
^y\ we choose exactly one number {i.e. location) in the 
program to modify before the user initiates any changes to 
{rifx^j^rify^]). The results of this step are two univariate 
equations to solve. Second, we define a mouse trigger that in¬ 
vokes the solver with each equation and then combines their 
resulting substitutions. Once mouse triggers have been com¬ 
puted for all shapes, the editor is prepared to respond to any 
user action with a local update to the program. We will now 
describe each step in detail. 


Shape Assignments, Our task is to determine a shape as¬ 
signment 7 that maps each shape to an attribute assignment. 
We define an attribute assignment 0 to map attribute names 
{i.e. little strings) to program locations. We refer to the 
range of an attribute assignment as a location set. 

Let hoxi refer to each rectangle from sineWaveOf Boxes 
in left-to-right order. Using a procedure Locs to collect 
all non-frozen locations that appear in a trace, we see 
that the ^x^ and ^y^ attributes are each computed us¬ 
ing two locations: Locs(Tr(66>x4^x^])) = {xo^sep} and 
Locs(Tr( 6 ox 4 ^y^])) = {yo^ amp}. As a result, there are 
four possible attribute assignments for each shape: 

0i = [^x^ xo, 'y' ^ yo] 

^2 — [^X^ xo, ^y^ amp] 

O3 = ['x' sep, 'y' yo] 

O4 = sep^ ^y^ ^ amp] 


These assignments correspond to the four options (top-left, 
top-right, bottom-left, and bottom-right, respectively) de¬ 
picted in [Figure Tp . 


^‘FaiU^ and Other Heuristics, As described in[§^ our de¬ 
fault strategy is to choose an attribute assignment whose 
range {i.e. location set) has not yet been assigned to any 
other shape in the output canvas. When all possible as¬ 
signments have been chosen an equal number of times {i.e. 
when they have been treated “fairly”), then we arbitrarily 
choose. As a result, we “rotate” through each of the four at¬ 
tribute assignments, assigning j{boXi) = Oj, for all i where 
j = 1 + (i mod 4). 

The fair heuristic will not always make choices that the 
user would prefer best. However, we find that even simple 
heuristics such as this one already enable a large degree of 
desirable interactivity. Therefore, designing more sophisti¬ 
cated heuristics could be a fruitful avenue for future work. 
In Supplementary Appendices El, we describe a second 
heuristic that we have implemented, which “biases” towards 
program locations that are used in relatively few run-time 
traces and, thus, have fewer opportunities to be assigned to 
a zone. We will not discuss this alternative in detail, because 
the vast majority of the examples we have written to date, 
including the ones discussed in this paper, work at least as 
well using the fair heuristic. 


Computing Triggers, The next task is to prepare for when 
the user might click a boxi in the output and drag it dx pixels 
in the x-direction and dy pixels in the ^-direction. 

Let po be the mapping from locations to numbers in 
the original sineWaveOfBoxes program and let 70 be the 
shape assignment computed using the heuristics described 
above. For each boxi, we evaluate the helper procedure 
ComputeTrigger(po5 705 boXi), where SolveOne is a solver 





that is given exactly one univariate equation to solve: 

ComputeTrigger(/), 7, i;) = 

X{dx, dy). [lx ^ SolveOne(/), ^x^ = tx)) 

0 {iy i-> So\yeOr\e{p,iy,ny = ty)) 

where nx^^=v[^x^] = Ux + dx ix = 

ny*y=v[’y’] n'y = ny + dy £y=y{v){’y’) 

When the user drags some boxi, its new attribute values n'^ 
and n'y (directly manipulated by the user) are used to solve 
the value-trace equations using the locations assigned by 
7 ( 1 ;). This substitution is then applied to the original pro¬ 
gram, the new program is run, and the new output is ren¬ 
dered as the user moves the mouse. When the user releases 
the mouse button, we compute new shape assignments and 
mouse triggers in anticipation of the next user action. 

Recap: Design Decisions, There are two aspects of our 
approach that warrant emphasis. The first that is we choose 
exactly one location to modify per updated attribute, even 
though there may be additional solutions {i.e. local updates) 
that modify multiple locations. For example, [Equation 3^ can 
also be satisfied by the substitution pq[xo •—> Sb][sep 20 ] 
(among many others). By considering only “small” local 
updates, however, we reduce the space of possible updates 
to synthesize and choose from. 

The second is that our solutions are only plausible, not 
faithful, because the same location may appear in multiple 
attributes being directly manipulated (and, therefore, multi¬ 
ple equations). For example, consider the box generated by 
the following expression: 

(let xy 100 (rect ^red^ xy xy .) ... 

The attribute assignment [ xy, 1 -^ is the only 

one to consider, but the corresponding system of con¬ 
straints on xy is overconstrained; the new values com¬ 
puted by SolveOne(p, xy, 100 ^ dx = xy) will differ from 
SolveOne(p, xy, 100 dy = xy) whenever dx 7 ^ dy. We 
could choose to apply an update only when the individual 
solutions agree, or, more conservatively, disallow the shape 
from being manipulated at all. Instead, we simply apply 
the individual substitutions in an arbitrary (implementation- 
specific) order, which has the effect of satisfying at least one 
of the constraints imposed by the user action. This approach 
trades synthesizing only faithful updates in exchange for ad¬ 
ditional opportunities to directly manipulate output values. 

4.2 Other Shapes and Zones 

For the purposes of presentation, so far we described a single 
type of user action, namely, dragging the interior of a rectan¬ 
gle. In practice, there are many other kinds of user actions. 
For each kind of SVG shape, we define zones that identify 
and name particular visual areas of a shape that can be di¬ 
rectly manipulated by the user in order to affect particular 


attributes. The screenshot below depicts zones for several 
kinds of shapes. 

As we have described, dragging the 
Interior zone of a rectangle allows the 
user to manipulate its and ^y^ at¬ 
tributes. Not all zones are tied to ex¬ 
actly two attributes, however. For exam¬ 
ple, the rectangle RightEdge zone is 
tied to one attribute (^ width ^ and the 
BotLeftCorner zone is tied to three 
(^x\ ^widths and ^height0- Furthermore, not all at¬ 
tributes vary covariantly with dx or dy. For example, when 
the user manipulates the BotLeftCorner of a rectan¬ 
gle, the ^ width ^ attribute varies contravariantly with dx 
(and, at the same time, ^x^ varies covariantly). Neverthe¬ 
less, the approach we described for assigning triggers for 
Interior zones generalizes in a straightforward way to the 
remaining shapes and zones. One slight change is that shape 
assignments are indexed by shape and zone, for example, 
7 ( 1 ;) (Interior) ( ^x^). We provide more details in Supple¬ 
mentary Appendices ca. 

5 . Implementation 

We have implemented Sketch-n-Sketch (available at 
http://ravichugh.github.io/sketch-n-sketch) in 
approximately 6,000 lines of Elm ca and JavaScript code. 
When the user hovers over a zone, our implementation dis¬ 
plays a caption that indicates whether the zone is “Inactive” 
or “Active” and, for the latter, identifies the constants {i.e. 
location set) that will change if the user manipulates it. Fur¬ 
thermore, we highlight these constants in yellow before the 
user begins manipulating the zone; in green while they are 
being updated during manipulation; and in red if the solver 
fails to compute a solution based on the user’s update. We 
use gray to highlight constants that contributed to an at¬ 
tribute value but were not selected by the heuristics. 

In the rest of this section, we describe the simple value- 
trace equation solver that we currently use and we evaluate 
the overall interactivity of our tool. In Supplementary Ap¬ 
pendices ina, we describe additional features of our imple¬ 
mentation. 

5.1 Solving Value-Trace Equations 

The mouse triggers defined in the previous section require a 
procedure SolveOne(p, I, n' = t) that, given the substitution 
p from the previous program and a location £, computes a 
new value for £ that satisfies the equation n' = t. Currently, 
we implement a simple solver that supports only “single¬ 
occurrence” equations, where the location i being solved 
for occurs exactly once. Our top-down procedure uses the 
inverses of primitive operations to recursively solve a uni¬ 
variate equation in a syntax-directed manner (see ifTSl for 
details). Not all primitive operations have total inverses, so 
SolveOne sometimes fails to compute a solution. 







As we will discuss below, supporting this syntactic class 
of equations is already enough to enable program synthesis 
for a variety of interesting examples. Our solver is easy to 
implement and deploy in our Web-based setting and fast 
enough to provide interactivity. Future work, however, may 
incorporate more powerful solvers (such as MATLAB or 
Z3 ifTHl ) while taking care to ensure that synthesis is quick 
enough to incorporate into an interactive, portable, direct 
manipulation editor. 

5.2 Interactivity 

The goal of Sketch-n-Sketch is to provide immediate, 
live synchronization updates in response to direct manip¬ 
ulation changes. For a user action to be “successful” re¬ 
quires that the particular zone be Active, that the solver com¬ 
putes an update in response to the mouse manipulation, and 
that the resulting update is applied to the program and re¬ 
evaluated within a short period of time. We discuss each of 
these aspects in turn based on measurements collected from 
68 little programs of varying complexity, spanning more 
than 2,000 lines of code in total. Below, we discuss summary 
statistics across all examples; for reference, detailed tables 
can be found in ifTSll . 

5.2.1 Active Zones 

For any particular zone, our assignment algorithm may con¬ 
sider zero, one, or more candidate location assignments 
based on the traces of its attributes. A zone is Inactive when 
there are zero candidates and is Active otherwise. Across 
all of our examples, there were a total of 3,772 shapes with 
14,106 zones, of which 991 (7%) were Inactive and 13,115 
(93%) were Active. 


Zones 

14,106 


Inactive 

991 

7% 

Active 

13,115 


Unambiguous 

4,856 

34% 

Ambiguous 

8,259 

59% 


Ambiguity, Among Active zones, 4,856 (34% of all zones) 
had exactly one candidate location assignment and 8,259 
(59% of all zones) had more than one (3.83 candidates on 
average). To provide responsive interaction, it is important 
to deal with ambiguities because they are so frequent. Our 
heuristics resolve ambiguities without user intervention. It 
may be fruitful to explore other approaches, such as showing 
multiple options for the user to choose from (particularly 
when there are relatively few), or allowing the user to make 
multiple user actions before attempting to infer an update. 

5.2.2 Solving Equations 

Next, we evaluate the solvability of equations that corre¬ 
spond to Active zones. Consider a program with initial lo¬ 
cation substitution p and shape assignment 7, and a shape 


V with an active zone C- For each attribute that C con¬ 
trols, 7 (i;)(C)( ^) identifies a location £ to update in order 
to solve the equation n -f d = f, where is the original 
value of ^[^k^], £ is one of the locations in f, and d is the 
change dictated by a user action. Across all examples, there 
are 28,222 such {p,vX^£^n,t) tuples. Because traces are 
often shared by multiple shapes and zones, we filter out tu¬ 
ples that are identical modulo v and leaving 4,574 unique 
(p, £, n, t) tuples. In the following, we refer to each of these 
tuples as a “pre-equation.” 


Unique Pre-Equations 

4,574 


Outside Fragment 

919 

20% 

Inside Fragment 

3,655 


No Solution for d = 1 

194 

4% 

Solution for d = 1 

3,461 


No Solution for d = 100 

438 

10% 

Solution for d = 100 

3,023 

66% 


Syntactic Fragment, The majority of pre-equations (3,655, 
which constitutes 80%) fall into the syntactic fragment han¬ 
dled by our solver. We paid little attention to the structure 
of traces when writing examples, so we have been surprised 
that this number is so high. We fully expected to incorpo¬ 
rate a more full-featured solver early in our work, but we 
have been able to leave this to future work without severely 
hampering the examples we have written so far. 

The remaining 919 (20%) pre-equations fall outside the 
fragment and are guaranteed not to be solvable. Our current 
attribute assignment algorithm does not take this into con¬ 
sideration and will sometimes assign such pre-equations to a 
zone. It would be worthwhile to avoid making such choices 
in the future. 

Solvability, For each pre-equation {p,£,n,t), we would 
like to know whether the solver can compute an update if the 
user manipulates the given attribute to be n + d. Rather than 
symbolically analyzing the space of possible user changes, 
we tested SolveOne(p, n + d = t) with two concrete 
values, namely, d = 1 and d = 100. Of the 3,655 pre¬ 
equations in the fragment, 3,461 were solvable for d = 1 
(Le. a green highlight) and the remaining 194 (4% of all 
unique pre-equations) were not {i.e. a red highlight). Note 
that simply computing an update does not necessarily mean 
that the change is acceptable to the user. 

Of the 3,461 pre-equations solvable for d = 1, 3,023 
(66% of all unique pre-equations) were also solvable with 
d = 100. The remaining 438 (10% of all pre-equations) were 
not. Upon inspection, several of these equations are of the 
form n d = f{cos£), where / is some function of cos^. 
Because the cosine function is bounded to the range [—1,1], 
the equation does not always have a solution. Indeed, there is 
a mismatch between the interpretation of user updates in the 
Cartesian plane and attributes like rotation that have more 
natural representations in other coordinate systems. In our 








experience, we have found that manipulating rotation angles 
in Sketch-n-Sketch often works better with explicit slid¬ 
ers or using separate built-in rotation zones in our implemen¬ 
tation, which we have not described in the paper. 

5.2.3 Performance 

In our experience, Sketch-n-Sketch is responsive for 
many, but not all, of our examples. We have not attempted 
to measure the observed frame rate of Sketch-n-Sketch, 
which depends on several factors beyond our implementa¬ 
tion. We have, however, measured the performance of four 
critical aspects of our implementation: parsing and evalu¬ 
ating a program, preparing for a user action, and solving 
a pre-equation. We performed our experiments on an Intel 
Core i7 (four cores, 2.6-GHz) running Mac OS X 10.9.5. For 
“Parse,” “Eval,” and “Prepare,” we tested the operation five 
times on every example using Firefox 45 and five times on 
every example using Chrome 49. For “Solve,” we tested the 
operation on Chrome 49 twice per pre-equation across all ex¬ 
amples. The “Min” and “Max” columns report the minimum 
and maximum times across all runs; “Med” and “Avg” report 
the median and average across all runs. Detailed statistics by 
example may be found in d. 


Operation 

Min 

Med 

Avg 

Max 

Parse 

9 ms 

53 ms 

77 ms 

520 ms 

Eval 

<1 ms 

5 ms 

12 ms 

165 ms 

Prepare 

1 ms 

13 ms 

200 ms 

6,789 ms 

Solve 

<1 ms 

<1 ms 

<1 ms 

14 ms 


As the user drags the mouse during direct manipulation, 
Sketch-n-Sketch repeatedly solves the trace equations 
for the zone being manipulated and re-evaluates the program 
to immediately display the interaction results. The average 
time to “Solve” each trace equation is negligible, <1 ms on 
average, because our solver uses a simple, syntax-directed 
procedure. Re-evaluation takes longer, 12 ms on average. 
Our implementation re-runs the entire program even though 
much of the output may not change. In the future, it would 
be useful to optimize the implementation to recompute only 
the parts of the program needed (e.g. GD). 

The slowest operations reported above, “Parse” and “Pre¬ 
pare,” are not run during direct manipulation. “Prepare” en¬ 
capsulates the computation of both shape assignments and 
triggers for all zones. We only perform this computation 
when the program is run initially and after the user finishes 
dragging a zone. Some of the data structures and algorithms 
we use for computing candidate location assignments and 
choosing from among them are rather naive and can be opti¬ 
mized in the future. 


cally defined graphics and direct manipulation. The imple¬ 
mentations resemble typical programs in other functional 
languages, but for the domain of SVG. 


6.1 Programmatic Abstractions 

Our current implementation does not allow new shapes to be 
added directly using the GUI. Nevertheless, we have used 
Sketch-n-Sketch to effectively program and manipulate 
several designs that would be difficult to edit or maintain 
using existing direct manipulation tools such as Illustrator 
and PowerPoint. Figure 3 provides thumbnails for some of 
the examples we will discuss. 


Variables as Abstractions, Sketch-n-Sketch does not 
attempt to infer any abstractions. It only propagates ab¬ 
stractions that result from shared constants in the program. 
Therefore, our little programs are structured to use vari¬ 
ables (bound to constants) to encode explicit relationships 
between attributes. Once these relationships have been de¬ 
fined, the Sketch-n-Sketch editor preserves them dur¬ 
ing direct manipulation. Many examples benefit from using 
variables as abstractions, such as: our Sketch-n-Sketch 
logo, which comprises three black polygons evenly spaced 
by white lines; the logo for the Chicago Botanic Gar¬ 
den (www.chicagobotanic.org), which contains several 
Bezier curves reflected across a vertical axis; the Active 
Transportation Alliance logo (www.activetrans.org), 
which uses several points along a path to depict a city sky¬ 
line; and a logo adapted from the Lillicon |01 project, where 
several curves are used to define a semi-circle. For each 
example, a single direct manipulation update changes all 
related attributes, without the need for any secondary edits. 


Derived Shapes, It is useful to define abstractions on top of 
the primitive SVG shapes. We define an nStar function (and 
include it in Prelude) that creates an n-sided star centered at 
(cx, cy) and rotated rot radians in the clockwise direction, 
where the distance from the center to the outer points is lenl 
and the distance to the inner points is len2. 


(def nStar 

(A(fill stroke w n lent len2 rot cx cy) ...)) 


We use nStar to implement the City of Chicago flag, which 
contains four evenly-spaced six-sided stars. By directly ma¬ 
nipulating the Point zones of a star in live mode, we can 
control the outer and inner distances of all four stars. Mod¬ 
ifying length parameters this way can be surprising. For ex¬ 
ample, using negative lengths leads to interesting patterns, 
even though one might not think to try them when program¬ 
ming without immediate visual feedback. 


6. Examples 

We have used Sketch-n-Sketch to implement a variety 
of designs. In this section, we will highlight observations 
that pertain specifically to the combination of programmati¬ 


Group Box Pattern, We occasionally find it useful to cre¬ 
ate a transparent rectangle in the background with the width 
w and height h of an entire design. Then, the BotRight- 
CORNER zone of this box will, predictably, be assigned the 
location set /i}. If we define all other shapes relative to 







Figure 3. Examples (left to right): Sine Wave of Boxes, Sketch-n-Sketch Logo, Chicago Botanic Garden Logo, Active 
Transportation Alliance Logo, Icon from Lillicon (41, City of Chicago Llag, Hilbert Curve Animation 


w and h, we gain direct manipulation control over the size of 
the entire design. In future work, it may be useful to provide 
built-in support for grouping shapes. 

Dealing with Ambiguities. We often start programming 
a design with all constants unfrozen except those that 
are not design parameters, such as 2 in the expression 
(* 2! (pi)). Then, after seeing how direct manipulation 
induces changes, we edit the program to freeze some con¬ 
stants. Linally, to deal with any remaining undesirable auto¬ 
matic choices from the heuristics, we add range annotations 
to certain numbers so that we can unambiguously and easily 
manipulate them with sliders instead. 

We performed a preliminary study (described in Supple¬ 
mentary Appendices El) that demonstrates the existence 
of scenarios (A) where using sliders is preferable to relying 
on heuristics for disambiguation, and (B) where relying on 
heuristics is preferable to using sliders. A systematic user 
study would be a useful direction for future work. 

‘Animations.’^ In several examples, like for the rendering 
of Hilbert curves, we use sliders to control the SVG design 
as a function of a numeric parameter. The effect is that we 
can “animate” the design as we manipulate the slider. In the 
future, we plan to support dynamic, time-varying animations 
as language and editor primitives. 

Procedural vs. Relational Constructions. There is a trade¬ 
off between procedural programming (in a functional lan¬ 
guage like little) and constraint-oriented programming 
(in a system like SketchPad (321). A detailed comparison of 
programming graphic designs in these two styles may be an 
interesting avenue for future work. 

6.2 Detailed Case Study: Ferris Wheel 

To provide a sense of when direct manipulation works 
smoothly, and how to deal with situations when it does not, 
we discuss one of our examples in more detail. In this de¬ 
sign, we manipulate a ferris wheel comprising a number of 
equal-length spokes emanating from a central hub, each of 
which has a passenger car at its end. 

Phase 1: Initial Development. In [Figure 4| \, 

we define a little program that embodies our 

design; for now, ignore the parts typeset in boxed V . ® ) 

blue. We define several parameters on lines 1 and 

2: the center (cx, cy) of the wheel; the number numSpokes 

and length spokeLen of the spokes; the radius rCenter for 

the central disc; the width wCar of each passenger car; the 


radius rCap of each car’s hubcap; and the rotation rot Angle 
for the entire design. We draw several components of the 
wheel on lines 4 through 11 using circles and rectangles, and 
we draw the spokes in terms of the nStar function described 
earlier. The cars are defined so that they remain vertical 
even when the wheel is rotated, in order to accurately portray 
the physical characteristics of a ferris wheel in motion. The 
visual rendering of the output is shown above. 

Phase 2: Direct and Programmatic Edits. 

Suppose we wish to edit the program so that 
the output resembles the picture on the right. 

In particular, we will adjust the size and lo¬ 
cation of the wheel, the size of the passenger 
cars, the number of spokes, the rotation angle, and the color 
of the first car. These changes will require a combination of 
programmatic and direct manipulation edits. 

First, we want to change the size and location of the 
wheel. When we hover over the Interior of the rim, 
Sketch-n-Sketch shows a caption to indicate that cx and 
cy will be updated. When we hover over the Edge of the 
rim, we see that spokeLen will be updated. In other words, 
Sketch-n-Sketch has chosen the following assignments: 

(rim, Interior) cy] 

(rim , Edge) I— spokeLen] 

These are, in fact, the only choices that could have been 
made, because the traces for the relevant attributes were 
atomic locations. Dragging these zones makes it easy to 
adjust the location and size of the overall design. 

Next, suppose we want to change the size of the passen¬ 
ger cars. The ^ width^ of each rectangle is defined by a 
single location, wCar. Therefore, the assignment maps the 
RightEdge of every car to wCar\ 

(carsi, RightEdge) i-^ [^width^ i— wCar] 



Dragging any of these RightEdge zones allows us to easily 
change the ^ width ^ of all cars. 

Now, suppose we want to change the number of spokes 
and the rotation angle. When hovering over the INTE¬ 
RIOR of several cars, we see that, based on the heuristics, 
Sketch-n-Sketch has chosen to vary numSpokes and 
rot Angle for several cars. 


(carso, Interior) 
(carsi , Interior) i-^ 
(cars2, Interior) i-^ 
(cars3, Interior) 
(cars4 , Interior) 


[^x,y^ 1-^ wCar] 
[^x,y^ 1-^ numSpokes] 
[^x,y^ 1-^ rot Angle] 
[^x,y^ 1-^ spokeLen] 
[^x,y^ 1-^ numSpokes] 

























(A) Initial f errisWheel. little program in black and manual code edits in [boxed blue 


(def [cx cy spokeLen rCenter wCar rCap] [220 300 80 20 30 7]) 
(def [numSpokes rotAngle] [5| ! {3-15} I 0| ! {-3.14-3.14} 1] ) 


(def ferrisWheel 

(let rim [(ring Markgray^ 6 cx cy spokeLen)] 

(let center [(circle ^black^ cx cy rCenter)] 

(let frame [(nStar ^transparent^ Markgray^ 3 numSpokes spokeLen 0 rotAngle cx cy)] 
(let spokePts (nPointsOnCircle numSpokes rotAngle cx cy spokeLen) 


(let cars 
(let hubcaps 


(mapi (A[i [x y]] (squareCenter 
(map (A[x y] (circle ^black^ x y rCap)) spokePts) 


(if (= 0 i) ^pink^ ^ light gray ^ jT] x y wCar)) spokePts) 


(concat [rim cars center frame hubcaps]) ))))))) 


(svg ferrisWheel) 


(B) Traces for the and ^y ^ attributes of the five ^rect ^ cars: 

CARx{i) = HUBCAPx{i) — (wCar I2) HUBCAPx{i) = ca: + spokeLen * cos((7r/2) — rotAngle + 2 * tt * (i /numSpokes)) 
CARy{i) = HUBCAPy{i) — (wCarI2) HUBCAPy{i) = ey — spokeLen * sin((7r/2) — rotAngle + 2 * tt * (i /numSpokes)) 


Figure 4. Ferris Wheel Example in Sketch-n-Sketch 


Dragging some of the cars has strange effects. To under¬ 
stand why, consider the traces of their ^x^ and ^y^ at¬ 
tributes, shown in [Figure 4^ ; we have simplified the traces 
slightly (using constant folding) and displayed them using 
infix notation to improve readability. The sines and cosines 
that appear in the traces come from the nPointsOnCircle 
library function, which we use to position the cars at the 
end of each spoke. If we drag carsi or cars 4 , the updated 
value for numSpokes is approximately 0 .3, which has the 
unintended effect of changing the number of spokes. In fact, 
this is an example where condition (c) of the definition of 
plausible updates is not satisfied; the new program does not 
compute an output value that is structurally equivalent to the 
original. If we drag cars 2 , rotAngle is updated but the ro¬ 
tation is not smooth and intuitive (this kind of equation was 
discussed in |§5.2.2| ). So, we use the editor’s Undo feature to 
restore the original values of numSpokes and rotAngle. 

Because we cannot easily manipulate the numSpokes and 
rotAngle parameters, we annotate them with ranges on line 
2; these changes are depicted with blue boxes. Furthermore, 
we annotate them as frozen so that no direct manipulation 
zones (such as the INTERIOR ones for cars) change these 
values. Instead, we rely on the sliders to control them. 

Finally, suppose we want to change the color of the first 
car, which will make it easier to observe how the wheel is ro¬ 
tated. Currently, Sketch-n-Sketch does not infer updates 
that introduce new control-flow into the program, so we edit 
the expression on line 9 to choose a different color for the 
car with index 0 . As a result of our programmatic edits, di¬ 
rect manipulation, and indirect manipulation via sliders, the 
output of our final program resembles the image at the be¬ 
ginning of this section. Furthermore, having identified what 


changes are easy to make with direct manipulation and what 
changes to make via sliders, we can quickly make subse¬ 
quent changes to the design parameters. 


6.3 Helper Value Design Pattern 

The sliders provided by Sketch-n-Sketch, which we re¬ 
fer to as user interface widgets, are similar to the notions of 
instruments m and surrogate objects 1 ^ . both of which 
aim to provide GUI-based control over attributes that are 
not traditionally easy to directly manipulate 1291 . Next, we 
show how to derive custom user interface widgets directly 
in little. Our key observation is to implement “helper” 
shapes whose attributes affect other parameters of interest. 


User-Defined Widgets, Suppose we 
are unhappy with the sliders built-in 
to Sketch-n-Sketch (|§2.41). We can 


3.1415 



design our own in little, which are 
used by the program below and de¬ 
picted in the adjacent screenshot. One 
slider controls a floating-point number n, one controls an in¬ 
teger i, and two control booleans bl and b 2 . 


(def [n si] (numSlider ... 0! 5! ^ 3.1415^0) 

(def [i s2] (intSlider ... 0! 5! ^i = ^ 3 . 1415 ^ 2 )) 

(def [bl s3] (boolSlider ... ^bl = ^ 0.25^U) 

(def [b2 s4] (boolSlider ... ^b2 = ^ O.TS^U) 


Directly manipulating the sliders indirectly manipulates the 
constants at locations ^i, ^ 2 , and ^4 (and, hence, the 
values bound to n, i, bl, and b 2 ). 

Both numSlider and intSlider are defined in terms of 
a slider helper function: 


(def slider (A(round xO xl y min max s src) ...)) 














The former returns src clamped to the range [min, max], if 
necessary; the latter, furthermore, rounds src to the near¬ 
est integer. We refer to the number supplied as the src 
parameter to be the “source” number (or “seed”) used to 
derive the “target” value, which is the first element of the 
pair returned by slider. The second element of the pair is 
the list of shapes that comprise its visuals. The idea is to 
place a “ball” on the line between (xO ,y) and (xl ,y) at a 
distance proportional to (src - min) / (max - min). 
The editor provides a button for hiding shapes marked 
with a special ^HIDDEN^ attribute, which we add to these 
helper shapes. We employ the same approach to implement 
boolSlider for directly manipulating booleans. In partic¬ 
ular, a boolSlider is tied to a source value between 0.0 
and 1.0, where values less than (resp. greater than) 0.5 
represent true (resp. false). 

Rounded Rectangles, The zones sup¬ 
ported by Sketch-n-Sketch control 
only the primary attributes for each SVG 
shape kind {e.g. ^x\ ^y\ ^width\ and 
^height ^ for rectangles). By combining user-defined slid¬ 
ers and the thin wrapper around the full SVG specification 
language, it is easy to write a little function that abstracts 
over additional parameters, such as ^ rx ^ and ^ ry ^ for spec¬ 
ifying rounded corners, and draws sliders (scaled based on 
the primary attributes) next to the rectangle to control them. 

(def roundedRect (A(fill x y w h rx ry) ...)) 

Tile Pattern, Our last example demonstrates how custom 
UI widgets can control more than just individual parameters. 
In the screenshot below, the left (resp. right) half shows 
the canvas with helper shapes displayed (resp. hidden). We 
employ three new kinds of helper shapes in this design. 
First, xySlider is 
a “two-dimensional” 
slider that allows the 
control of two param¬ 
eters simultaneously. 

In this example, we 
draw the xySlider 
directly atop the grid. 

Dragging its handle, the black circle in the lower-right cor¬ 
ner, around the grid provides an intuitive way to change the 
number of rows and columns. Next, we use enumSlider 
(drawn above the grid) to select from a list of different 
shapes. Then, we define red circles (to the left of the grid) 
to be “tokens” that denote “selection” when dragged over 
particular tiles in the grid. We define a helper function 
isCovered to check whether any token is currently placed 
over the tile centered at (cx,cy). Once we are done using 
these helper objects to manipulate the grid, we use the built- 
in editor feature to toggle the visibility of helper objects, 
leaving us with the final design shown in the right half of 
the screenshot above. 



> : 



Recap: Customizing the UI, Sketch-n-Sketch could 
provide built-in support for some of the helper objects we 
described (custom sliders and rounded rectangles). However, 
no matter how many features are built-in, we believe there 
will always be situations where a custom tool would be a 
better fit for the task at hand. With prodirect manipulation, 
the user can push the frontier beyond what is provided. 
Exploring this boundary between primitive and custom UI 
widgets may be fruitful, both for designing useful libraries 
as well as motivating new built-in features. 

7. Discussion 

In this paper, we presented an approach for live synchro¬ 
nization of a program and changes to its output, by in¬ 
strumenting program evaluation to record run-time traces, 
phrasing user updates in terms of a new framework called 
trace-based program synthesis, and designing heuristics to 
automatically resolve ambiguities. One may think of pro¬ 
grams in our approach as sketches (in the program synthesis 
sense 1301 ) where the holes are numeric constants, and the 
requirements for filling holes (i.e. changing numbers) come 
from the sketches (in the drawing sense) in the graphical user 
interface. Hence the name Sketch-n-Sketch. 

7.1 Related Work 

In a recent position paper (121, we provided a broad overview 
of relevant program synthesis (e.g. (231 [24l |3T1), program¬ 
ming by example (e.g. EESlEa), and bidirectional pro¬ 
gramming techniques (e.g. CS). Here, we focus our discus¬ 
sion on projects related to vector graphics. 

Several projects use programming languages, direct ma¬ 
nipulation interfaces l29l , or some combination to provide 
expressive means for manipulating visual output. We clas¬ 
sify them using the following interaction modes identified 
by Bret Victor in a talk on drawing tools Cl: “Use” for using 
built-in functionality through menus and buttons; “Draw” for 
directly manipulating domain objects; and “Code” for writ¬ 
ing programs that manipulate domain objects. 

Dynamic Drawing (Use + Draw), Victor’s prototype in¬ 
teractive drawing editor (71, Apparatus (34l, and Program¬ 
ming by Manipulation (211 provide expressive direct manip¬ 
ulation capabilities that serve as a way to build programs in 
restricted, domain-specific languages. By design, these tools 
tend to prohibit or discourage the user from manipulating 
content via the “indirect” mechanism of code. 

Although this choice may be desirable for many appli¬ 
cation domains and end users, we believe there are limits 
to what can be accomplished using features and transforma¬ 
tions provided by any tool. Therefore, our work targets users 
who wish to work both via direct and programmatic manip¬ 
ulation (i.e. Draw -i- Code). 

Programs that Generate Graphics (Code), Processing O 
is a language and environment for generating visual output 





that has been popular both in classroom and commercial set¬ 
tings. Follow-on projects, such as Processing.]s (281, provide 
similar development environments for Web programming. 
These systems provide immediate and interactive output, but 
they do not provide ways to directly manipulate output in or¬ 
der to modify the program that generated it. 

GUIs that Generate Programs (Draw + Code). Graph¬ 
ical user interfaces (GUIs) for creating visual output in 
many domains often generate “code behind” what the user 
directly manipulates. Such tools include PaintCode Ezl, 
DrawScript ifBl . SVG-edit Oa, and Adobe Fireworks for 
graphic design. Programs generated by these tools, however, 
are typically just as low-level as the output itself, making 
them difficult to modify, maintain, and reuse. 

Constraint Programming (Draw + Code). Constraint- 
oriented programming systems, such as the classic Sketch- 
Pad (^ and ThingLab Q systems as well as their more 
recent incarnations ||6l[20l, are characterized by (i) declar¬ 
ative programming models that allow programs to specify 
constraints between program elements, and (ii) constraint 
solvers that attempt to satisfy these constraints, often using 
iterative and approximate numerical methods. Together with 
full-featured GUIs, SketchPad and ThingLab provide user 
experiences that tightly integrate programmatic and direct 
manipulation. 

Our goal is to support a similar workflow but for more 
traditional, deterministic programming models, which are 
used more regularly in a variety of domains. That is, we 
wish to factor all constraint solving to a program synthesis 
phase, rather than including it within the semantics of the 
programming language itself. 

Synthesis for Vector Graphics. The problem of beauti¬ 
fying user drawings has been long-studied in the graph¬ 
ics community and has recently been approached with 
programming-by-example techniques miiniiisi. These ap¬ 
proaches synthesize artifacts in domain-specific representa¬ 
tions and languages. 

In order to eliminate the need for secondary direct ma¬ 
nipulation edits, Lillicon (H synthesizes different represen¬ 
tations for the same graphic design based on the intended ed¬ 
its. In Sketch-n-Sketch, the user must pick a particular 
representation. But because this representation is a general- 
purpose program, we can often build abstractions that are 
preserved by prodirect manipulation, which avoids the need 
for secondary edits. 


7.2 Future Work 


We mentioned several ways to build on our work throughout 
the paper, including smarter heuristics and richer trace lan¬ 
guages that record control flow {e.g. il). We foresee several 
additional opportunities. 


Trace-Based Program Synthesis. There are several “knobs 
to turn” within the framework defined in[§^ The current for¬ 
mulation synthesizes updates given a run-time trace and a 
single updated value. In other settings, it may be fruitful to 
consider multiple traces, a history of user edits, and a history 
of previous program updates. Furthermore, it may be useful 
to rank candidate solutions according to the (soft) constraints 
not changed by the user. 


Live Synchronization for Other Domains. We plan to re¬ 
target our approach (language instrumentation, synthesis al¬ 
gorithm, and prodirect manipulation editor) to meet the spe¬ 
cific challenges of different domains, such as layout in text 
documents, formulas in spreadsheets, dynamic animations in 
presentations and data visualizations, and multiple rendering 
configurations for Web applications. 

Prodirect Manipulation. The vision of prodirect manipu¬ 
lation, which we identified in a position paper (121, com¬ 
prises three goals: (a) the ability to directly modify the out¬ 
put of a program and infer updates in real-time to match 
the changes (live synchronization); (b) the ability to synthe¬ 
size program expressions from output created directly via 
the user interface; and (c) the ability to temporarily break 
the relationship between program and output so that “larger” 
changes can be made, and then reconcile these changes with 
the original program (called ad hoc synchronization). 

We addressed the first goal in this paper. For the second, 
we plan to investigate ways to design direct manipulation 
operations that generate programmatic relationships. For the 
third goal, we plan to develop richer trace-based synthesis 
algorithms to infer larger, “structural” program updates, in 
contrast to the small, local updates we sought in this paper. 
These directions of future work will help fully realize the 
long-term vision of combining programming languages and 
direct manipulation interfaces. 


Acknowledgments 

The authors would like to thank Shan Lu, Kathryn McKin¬ 
ley, Loris D’Antoni, and the anonymous reviewers for valu¬ 
able suggestions that improved the final version of this paper. 



References 

[1] Daniel W. Barowy, Sumit Gulwani, Ted Hart, and Benjamin 
Zorn. FlashRelate: Extracting Relational Data from Semi- 
Structured Spreadsheets Using Examples. In Conference on 
Programming Language Design and Implementation (PLDI), 
2015. 

[2] Michel Beaudouin-Lafon. Instrumental Interaction: An Inter¬ 
action Model for Designing Post-WIMP User Interfaces. In 
Conference on Human Factors in Computing Systems (CHI), 
2000. 

[3] Ben Pry and Casey Reas. Processing. https:// 
processing.org/ 

[4] Gilbert Louis Bernstein and Wilmot Li. Lillicon: Using Tran¬ 
sient Widgets to Create Scale Variations of Icons. Transac¬ 
tions on Graphics (TOG), 2015. 

[5] Alan Borning. The Programming Language Aspects of 
ThingLab. Transactions on Programming Languages and Sys¬ 
tems (TOPLAS), October 1981. 

[6] Alan Borning and Bert Preudenberg. ThingLab. https: 
//github. com/cdglabs/thinglab 

[7] Bret Victor. Drawing Dynamic Visualizations, http:// 
worrydream.com/ 

[8] Satish Chandra, Emina Torlak, Shaon Barman, and Rastislav 
Bodik. Angelic Debugging. In International Conference on 
Software Engineering (ICSE), 2011. 

[9] Salman Cheema, Sumit Gulwani, and Joseph LaViola. Quick¬ 
Draw: Improving Drawing Experience for Geometric Dia¬ 
grams. In Conference on Human Factors in Computing Sys¬ 
tems (CHI), 2012. 

[10] Salman Cheema, Sarah Buchanan, Sumit Gulwani, and 
Joseph J. LaViola, Jr. A Practical Pramework for Constructing 
Structured Drawings. In International Conference on Intelli¬ 
gent User Interfaces (lUI), 2014. 

[11] Yan Chen, Joshua Dunfield, Matthew A. Hammer, and 
Umut A. Acar. Implicit Self-Adjusting Computation for 
Purely Punctional Programs. In International Conference on 
Functional Programming (ICFP), 2011. 

[12] Ravi Chugh. Prodirect Manipulation: Bidirectional Program¬ 
ming for the Masses. In International Conference on Soft¬ 
ware Engineering, Visions of 2025 and Beyond Track (ICSE 
V2025), 2016. 

[13] Ravi Chugh, Brian Hempel, Mitchell Spradlin, and Jacob 
Albers. Programmatic and Direct Manipulation, Together 
at Last (Supplementary Appendices), http: //arxiv. org/ 
abs/1507.02988 2016. 

[14] Leonardo de Moura and Nikolaj Bjprner. Z3: An Efficient 
SMT solver. In Tools and Algorithms for the Construction 
and Analysis of Systems (TACAS), 2008. 

[15] DrawScript. http://drawscri.pt 

[16] Evan Czaplicki. Elm. http://elm-lang.org 


[17] GNU Project. The gnu image manipulation program, http: 
//WWW.gimp.org/ 

[18] Sumit Gulwani. Automating String Processing in Spread¬ 
sheets Using Input-Output Examples. In Symposium on Prin¬ 
ciples of Programming Languages (POPL), 2011. 

[19] Sumit Gulwani, Vijay Anand Korthikanti, and Ashish Tiwari. 
Synthesizing Geometry Constructions. In Programming Lan¬ 
guage Design and Implementation (PLDI), 2011. 

[20] Hesam Samimi and Alex Warth. Sketchpadl4. http://www. 
cdglabs.org/sketchpadl4 

[21] Thibaud Hottelier, Ras Bodik, and Kimiko Ryokai. Program¬ 
ming by Manipulation for Layout. In Symposium on User 
Interface Software and Technology (UIST), 2014. 

[22] Zhenjiang Hu, Andy Schurr, Perdita Stevens, and James E. 
Terwilliger. Dagstuhl Seminar on Bidirectional Transforma¬ 
tions (BX). SIGMOD, 2011. 

[23] Etienne Kneuss, Viktor Kuncak, Ivan Kuraj, and Philippe 
Suter. Synthesis Modulo Recursive Eunctions. In Confer¬ 
ence on Object-Oriented Programming Languages, Systems, 
and Applications (OOPSLA), 2013. 

[24] Viktor Kuncak, Mikael Mayer, Ruzica Piskac, and Philippe 
Suter. Complete Punctional Synthesis. In Conference on 
Programming Language Design and Implementation (PLDI), 
2010 . 

[25] Bum Chul Kwon, Waqas Javed, Niklas Elmqvist, and Ji Soo 
Yi. Direct Manipulation Through Surrogate Objects. In 
Conference on Human Factors in Computing Systems (CHI), 
2011 . 

[26] Mikael Mayer, Gustavo Soares, Maxim Grechkin, Vu Le, 
Mark Marron, Oleksandr Polozov, Rishabh Singh, Benjamin 
Zorn, and Sumit Gulwani. User Interaction Models for Dis¬ 
ambiguation in Programming by Example. In Symposium on 
User Interface Software and Technology (UIST), 2015. 

[27] PaintCode. http://www.paintcodeapp.com 

[28] Processing.]s. http://processingjs.org/ 

[29] Ben Shneiderman. Direct Manipulation: A Step Beyond Pro¬ 
gramming Languages. Computer, August 1983. 

[30] Armando Solar-Lezama. Program Synthesis by Sketching. 
PhD thesis, UC Berkeley, 2008. 

[31] Saurabh Srivastava, Sumit Gulwani, and Jeffrey S. Poster. 
Prom Program Verification to Program Synthesis. In Sym¬ 
posium on Principles of Programming Languages (POPL), 
2010 . 

[32] Ivan Sutherland. Sketchpad, A Man-Machine Graphical Com¬ 
munication System. PhD thesis, MIT, 1963. 

[33] SVG-edit. https://code.google.eom/p/svg-edit/ 

[34] Toby Schachman. Apparatus, http://aprt.us/ 

[35] World Wide Web Consortium (W3C). Scalable Vector Graph¬ 
ics (SVG) 1.1 (Second Edition). http://www.w3.org/TR/ 
SVGll/ 


Supplementary Appendices for the PLDI2016 Paper 


A. Little 

The syntax of little was defined in |Figure ^ of[^ Expres¬ 
sions e include booleans b, strings s, floating-point numbers 
n, lists (encoded as cons cells, or pairs), functions, function 
application, let-bindings, case expressions, and applications 
of primitive operations. Patterns p are either variables or list 
patterns used for deconstructing values. 

Syntactic Sugar, The following are derived in terms of the 
core expression forms, as usual. 


(def p ei) 62 
(defrec p ei) 62 

(if 61 62 63) 

(A (pi • • -pm) e) 

(60 61 • • • Cm ) 

[61 • • • Cm 1 

[61 • • • Cm I 60 ] 

[pi • • • Pm ] 
[Pl • • • Pm I PO ] 


(let p 61 62) 

(letrec p 61 62) 

(case 61 (true 62) (false 63)) 
(Api ••• (A 

Pm 6)) 

( ( (.Cq 61 ) ' ' ' ) Cm ) 

E 61 • • • Cm I El 1 
E61I E---I E6^|6o]]] 

Epi • • • Pm I El ] 

Epil E---I Epmlpolll 


SVG Attributes. We represent SVG elements with three- 
element lists E^i t ’2 '^ 3 ], where the second element V 2 is 
a list E [rc/ei Wyi~\ • • • Vwkm ] of m key-value pairs. 
Below, we define \_WkWy^ ^ svgAttr to be the translation 
from key-value pairs in little to attributes in the target, 
XML-based SVG format. 

Essil “s = 

Is n] ^ “s = 

E ^points ^ pts] ^ “points = s”, pts ^pts s 
E^fill^ rgbal ^ “fill = s”, rgba ^rgba s 
Essil s G {^Z0NES^ ^HIDDEN 

E Irixi riyi^ • • •] ^pts '''nxi,nyi • • • '” 

InrUgnbria^ ^rgba ' rghairir, Tig , Ub , Tla) ' 

We provide a thin wrapper over the target format by al¬ 
lowing little programs to specify arbitrary attributes using 
strings (the first translation rule). In our current implemen¬ 
tation, we do not include units in the translation of numeric 
attributes (the second rule), so numbers are interpreted as 
pixels. We support specialized encodings for several SVG 
attributes in order to provide direct manipulation access to 
their constituent numeric values. For example, the points of 
a polygon or polyline may be specified as a list (the third 
rule), and color attributes may be specified as RGBA compo¬ 
nents (the fourth rule). We also encode path commands {i.e. 
the M ^ attribute) so as to access their data points and Bezier 
curve control points; we elide the details of this encoding. 
We use non-standard attributes ^ZONES^ and ^HIDDEN^ for 
specific purposes in Sketch-n-Sketch. So, we eliminate 
them when translating to SVG (the fifth rule). 


Semantics. Compared to the constant literal range annota¬ 
tions in [Figure^ we can extend the syntax to allow expres¬ 
sions in ranges, written { ei - 62 }. The changes required to 
evaluate range expressions to numbers and propagate them 
through the evaluation relation are straightforward, so we 


elide them. Also, compared to the version shown in Figure 2 


we define a more general version of the E-Op-Num rule: 


eo opm ei 

n = I {opm ni-’ rim) I t = {opm h-’ tm) 
(Cq Cl ’ ' ’ C772) 'IJ' ^ 


B. Live Synchronization for SVG 

We provide additional discussion of some topics from[§4] 


Zones and User Actions. Figure 5 shows the zones asso¬ 


ciated with each kind of SVG node, as well as the attributes 
that are affected by directly manipulating that kind of zone. 
Recall that in the definition of MouseTrigger for Interior 
zones of rectangles, the new value of the attribute is 

defined as + dx. Therefore, we say that varies co- 

variantly with dx (denoted by in [Figure 3] ). Similarly, 
varies covariantly with dy (written -i-dy). As shown in 
the table, some zone attributes vary contravariantly (written 
— dx and —dy)io match the expected physical behavior in 
the user interface. We refer to these changes in attribute val¬ 
ues as ojfsets. Note that in [Figure 3] we write ^ and ^hMo 
abbreviate ^width^ and ^height \ 


B.l Mouse Triggers 

We describe how to generalize the approach we described 
for computing triggers for INTERIOR zones. Compared to 
the discussion in[§^ a shape assignment 7 maps a shape and 
zone to an attribute assignment. 


Computing Triggers. Compared to the description in § 4 


the ComputeTrigger procedure is also indexed by the shape 
kind (the first argument). 

For each ^rect^ node in the output, we evaluate the 
helper procedure ComputeTrigger (Interior, 7o: boxi), 
where SolveOne is a solver that is given exactly one univari¬ 
ate equation to solve: 


ComputeTrigger(lNTERiOR, p, 7,1;) = 

X{dx, dy). (4 ^ SolveOne(p, 4 , = 4 )) 

0 (4 ^ SolveOne(p, 4 ,^y = 4 )) 

where =i;[^x'] 4 = 7 (i;)(Interior)(' x') 

Uy^y =v[^Y^] iy = j{v) (Interior) {^Y^) 

= Ux -\- dx n'y = Uy dy 

Notice how the shape assignment is indexed by value v and 
zone Interior in order to retrieve the attribute assignment, 
which determines which locations to update in response to 
the user’s changes to the ^x^ and values. 
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Figure 5. SVG Zones and Offsets 


To generalize the definition of ComputeTrigger, we add 
versions for each different kind of shape (the first argument). 
The table in |Figure 5] dictates which attributes are updated by 
manipulation each zone and for INTERIOR) and 
the offset to apply to their previous values (-\-dx and + dy 
above). For example, we compute triggers for the Left- 
Edge zone of ^ rect ^ as follows. Notice that dx affects two 
attributes, and that dy does not affect any. 

ComputeTrigger(LEFTEDGE, p, 7, i;) = 

A(dx, dy). p® SolveOne(p,= tx)) 

0 {ly, ^ SolveOne(p,^^,n^ = t^)) 

where Ux^'' = v[^y:^] = 7(i;)(LeftEdge)(^x^) 

lyj = 7('i;)(LEFTEDGE)('w') 

n'^ = Ux ^ dx n'^ = riyj — dx 

As another example, we compute triggers for the BotLeft- 
CORNER zone (abbreviated to BotLeft) as follows. Notice 


that dx affects the same two attributes as LeftEdge, and 
that dy affects ^height ^. 

ComputeTrigger(BOTLEFT,p, 7, i;) = 

A(dx, dy). {ix ^ SolveOne(p, ix^ = tx)) 

0 {£yj ^ SolveOne(p,-^^,n^ = t^)) 

0 (4 ^ SolveOne(p,4,n'^ = 4)) 

where nx^'^=v[^x^] 4 = 7('^)(BotLeft)( ^x^ ) 

= i;['w'] 4 = 7(i;)(BotLeft)('w') 

= v[’h’] 4 = 7('^)(BOTLEFT)('h') 

n'^ = Ux dx n'^ = riyj — dx 

n'h = nh- dy 

As a final example, we compute triggers for the RightEdge 
zone of ^ circle ^ nodes as follows. 

ComputeTrigger(RlGHTEDGE, p, 7, v) = 

X{dx, dy). p 0 (4 SolveOne(p, 4, = 4)) 

where nr^^=v[^r’] 

4 = 7(1;) (RightEdge) (' r 4 
n'^ = rir dx 

In our implementation, we generalize this approach to allow 
the user to click and drag anywhere on the Edge of a 
^circle^ or ^ellipse^ to manipulate the radius attributes. 

^‘Biased^^ Heuristic, Program locations that are used in rel¬ 
atively few run-time traces of the output have fewer opportu¬ 
nities to be assigned to a shape according to the fair heuris¬ 
tic. Therefore, we offer an alternative that is “biased” to¬ 
wards program locations that occur in relatively few run¬ 
time traces. In particular, we count the number of occur¬ 
rences that each location appears across all traces of the out¬ 
put canvas, which we use to compute a score for the location 
set as Score(4i, • • • ,4}) = Count(^i) x • • • x Count(4)- 
Then, when choosing among multiple attribute assignments, 
we pick the one with the lowest score, in order to favor loca¬ 
tions that appear relatively infrequently. In our experience, 
these two simple heuristics work equally well on many ex¬ 
amples. Next, we show an example where the biased heuris¬ 
tic is preferable. 

Consider the variation below of the sineWaveOf Boxes 
example from |§ 2[ Compared to the original version, the 
value xO ^, computed from xO and two new values a and b, 
is used as the “base” x-position. 

(def [a b] [0 0] ) 

(def [xOC [(+ xO (+ a (+ a (+ b b))))]) 

(let xi (+ xO^ (* i sep)) 



















According to the fair heuristic, each of the four locations in 
{xO, a, b, sep} get chosen equally often (i.e. for every fourth 
box in the pattern). Because the first three of these locations 
all contribute to the same initial base position, the effect 
is that dragging three out of every four boxes manipulates 
the base position and one out of every four manipulates the 
spacing between them. 

The biased heuristic makes different choices for this ex¬ 
ample. There are n occurrences of both xO and sep in the 
traces of all attribute values of all shapes, and there 2n oc¬ 
currences of both a and b. Therefore, the former two loca¬ 
tions are preferred by the biased heuristic over the latter two. 
Assuming that this choice is combined with a “rotating” as¬ 
signment, then alternating boxes are assigned xq and sep, 
and no boxes are assigned a or b. 

Although this example is contrived, we find biased heuris¬ 
tic to be useful in some examples (such as the Floral Logo 
in our project repository), where many control points are 
positioned and rotated relative to a common initial position. 


Solving Univariate Equations (I and II). First, we define 
jCi = Locs{ti) to be the set of non-frozen locations {i.e. 
constants) that occur in ti, each of which is a candidate 
“variable” to solve for. Then, our equation solver. Solve, 
attempts to solve each equation n- = ti by treating the 
location li as the single variable, and using the substitution 
po to define the values of all other locations referred to 
in ti. In general, a location may appear in multiple traces. 
As a result, solutions may provide different new values for 
a location, and any subsequent bindings will “shadow” or 
override previous ones. As a result, the solutions are only 
plausible. 

If we wanted to synthesize faithful updates, we could 
instead define the sets €[ = Ci\ {^j^i Cj) to be disjoint 
sets of locations. The resulting equations would then be a set 
of independent equations. We choose to forgo this approach, 
however, because it can be overly restrictive. For example, 
consider a common pattern found when programming with 
points along a circle of length len: 


B.2 Synthesis Algorithm 

In[p| we described our approach for decomposing a system 
of constraints (induced by manipulating SVG zones) into 
several univariate equations. And in |§ 5[ we described the 
simple solvers that we currently use. In order to provide 
a more detailed description of our overall implementation, 
now we describe both of these pieces together and present 
more details on the simple solvers. Our algorithm rests on 
three design principles: 

(I) Solve only one equation at a time, rather than 
simultaneously solving a system of equations; 

(II) Solve only univariate equations; and 

(III) Solve equations only in simple, stylized forms. 

The first two design decisions follow the decomposition 
approach described in [§^ The last follows the observation 
that even a simple solver facilitates a variety of interesting 
use cases ( [§6| ), while being fast enough to incorporate and 
easy to deploy in our Web-based setting. As we will discuss, 
we infer plausible updates rather than faithful ones. 

The Algorithm. Suppose the user manipulates the m val¬ 
ues through rim^^ to be n'^ through respec¬ 

tively, which leads to the set of m trace-value equations 
{n'l = ti, • • • , = tjn}. Furthermore, let po be the map¬ 

ping of locations to numbers in the original program. We 
compute a set of local updates using the algorithm below, 
which we will walk through to explain how it encompasses 
our three design decisions. 

SynthesizePlausible (/)05 {^i = G, * * * 5 = ^m}) = 

{ ^i)) I G X • • • X } 

where ki = So\ye{po,£i^n'i = ti), 

Li = Locs(fi), and Ci = Li. 


(let xi (+ cx (* len (cos (angle i)))) 

(let yi (+ cy (* len (sin (angle i)))) ...)) 


Taking the disjoint location approach, only cx can be varied 
for the value of xi, and only cy can be varied for yi. Ma¬ 
nipulating these parameters via direct manipulation is useful, 
but it would be nice to directly manipulate the len param¬ 
eter as well. As a result, in our current implementation, we 
choose to forgo faithfulness in favor of admitting more pos¬ 
sible updates for direct manipulation. 


Solving Simple Equations (III). Figure 6 shows how we 
combine two different solvers, one for “addition-only” equa¬ 
tions (where the only primitive operation is +) and one for 
“single-occurrence” equations (where the location £ being 
solved for occurs exactly once). These two kinds of stylized 
equations are already enough to enable program synthesis 
for a variety of interesting examples ( [§^ . Future work, how¬ 
ever, may incorporate more powerful solvers while taking 
care to ensure that synthesis is quick enough to incorporate 
into an interactive, portable, direct manipulation editor. 

Our first solver, SoIvca, handles traces where the only 
operator used is + {cf. the discussion of programming with 
integers. Such equations are easy to solve: count the number 
of occurrences c of the unknown variable, and divide the 
partial sum s by c. Note that this yields a number, not 
necessarily an integer, as output. 

Our second solver, Solvee, handles equations where there 
is exactly one occurrence of the unknown location variable 
being solved for. For equations like these, we can define a 
top-down, syntax-directed procedure that uses inverses of 
primitive operations. 

In practice, Solvee subsumes SoIvca on virtually all equa¬ 
tions encountered in our examples. 


Inverse Operations. Our “single-occurrence” equation 
solver. Solves, recursively reduces the goal equation at each 



(O) Overall Solver: 

Solve(y9, n = t) = k 

if SolveA(p, — t) — k or SolveB(p, — t) — k 


(A) “Addition-Only” Solver: 


SolveA(p, — t) — (n — s)/c, 

where (c, s) = WalkPlus(y9, t) 


WalkPlus(y9,^,^) = (1,0) 
WalkPlus(y9,^,/) = (0,y9^') 
WalkPlus(y9, (+ ti t2)) = (ci + C2, Si + S2) 
where (ci, si) = WalkPlus(y9, ti) 
(C2, S2) = WalkPlus(y9, 12 ) 


C. Implementation 


Our implementation provides several little and SVG fea¬ 
tures features beyond those described inP 


Prelude. We have implemented a small Prelude library 
of little functions, in approximately 500 lines of code, 
that is included in every program. Some Prelude functions 
support general functional programming, and some support 
SVG programming, in particular. 


Thawing and Freezing Constants. By default, all con¬ 
stants that are not frozen may be changed by the synthesis 
algorithm. We allow the user to choose the alternative, treat¬ 
ing all constants as frozen except those that are explicitly 
thawed (written n?) in the program. 


(B) “Single-Occurrence” Solver: 

So\yeB{p,i,n — t) — n 

SolveB(p,^,n = (opi ti)) = SolveB(p,^, lnv(opi)(n) = ti) 
SolveB(p,n = (op2 ^1^2)) = 

{ SolveB(p,lnvL(op 2 ,'ni)(n) = 12 ) if pti = ni 
SolveB(p,lnvR(op 2 , rr 2 )(n) = ti) if pt 2 — n 2 

lnv(cos)(n) = [(arccosn)] 
lnv(arccos)(n) = [(cosn)] 
lnvL(+,ni)(n) = [(-nm)] 
lnvR(+,n2)(n) = [[(- nn2)\ 
lnvL(-,ni)(n) = [[(+nni)] 
lnvR(-, n2){n) = {(-712 n)j 
lnvL(*,ni)(n) = [(/nm)] 
lnvR(*,n2)(n) = [[(/ nn2)] 
lnvL(/, ni)(n) = [[ (/ ni n) ] 
lnvR(/,n2)(n) = [(* 71712)] 


Figure 6. Simple Value-Trace Equation Solver 


step by using inverses of primitive operators. We define 
Inv(op) to return the inverse operator of unary operator op. 
We define lnvL(op, tii) (resp. lnvR(op, 712 )) to be a func¬ 
tion that denotes the inverse of binary operator op partially 
applied to the first argument 77 1 (resp. second argument 772 ). 

Not all primitive operations have total inverses, so our 
solver sometimes fails to compute a solution. For example, 
recall [Equation T] from |§ 2| and consider a situation where 
sep is assigned to change (instead of xq) when the user 
manipulates the -position to be 115; there is no solution 
for SolveOne(po 5 155 = (+xq Iq sep))) because 
the expression (+ 50 (* 0 sep)) evaluates to 0 for all 
choices of sep. 


Programming with Integers. The little language pro¬ 
vides a single, fioating-point number type. When program¬ 
ming with (non-negative) integers, however, the user may 
choose to use the library functions mult, minus, and div 
instead of the primitive multiplication, subtraction, and divi¬ 
sion operators, respectively. These library functions result in 
values with addition-only traces. For example, compared to 


Equation 3 the expression (+ xO (mult i sep) leads to 


the addition-only trace (+ xq (+ sep sep)) when i is 2. 


Color Numbers. In addition to specifying colors as strings 
or RGB A numbers, we allow colors to be specified using 
a non-standard color number, where the range 0 to 500 is 
interpreted as a spectrum of colors, including grayscale. For 
an SVG shape with a fill color number {i.e. [ ill^ 77]), 
our editor displays a slider right next to the object that allows 
direct manipulation control over the ^ f ill ^ attribute. 


Layers. Our implementation currently provides two kinds 
of layers. We allow the user to toggle between displaying and 
hiding the zones associated with each shape. We also allow 
the user to toggle between displaying and hiding shapes that 
contain the non-standard ^ HIDDEN ^ attribute. We have found 
hidden shapes to be useful in several of our examples (|§6.3|). 


Exporting to SVG. In order to facilitate interoperation 
with other SVG editing tools, we provide an option to print 
the resulting canvas in SVG format (rather than rendering 
it), which can then be copied and pasted into other tools. 


D. Examples 

Compared to our discussion in |§ 6| here we provide more 
low-level details of some of our little examples. All of 
our examples are available on the Web[^ 

Boxes. The threeBoxesInt _ 

program is our “hello world” ex- ~ 

ample for prodirect manipulation. 

The number of boxes and their lo- _ 

cation, width, and height are sim- 

^ http://ravichugh.github.io/sketch-n-sketch 























pie parameters to change in the program. In addition, the lo¬ 
cation, width, and height can easily be changed in the direct 
manipulation editor. When manipulated in live mode, all of 
the boxes are updated together in real-time. The screenshot 
on the right shows the zones displayed to the user. 

Elm The Elm logo is a tangram consisting of seven 

polygons. We implemented this logo by massaging the defi¬ 
nition from the SVG format to the representation in little. 
This process will be automatic once we add support for im¬ 
porting SVG images directly. 

There are two noteworthy aspects of this 
example in Sketch-n-Sketch. The first is 
that our definition uses the WiewBox^ at¬ 
tribute to define a local coordinate system 
for shapes within the canvas. Because of 
the thin wrapper around SVG, the result in 
Sketch-n-Sketch is that the output canvas is scaled to 
fit the size of the canvas, no matter how large or small the 
application window it is. As a result, even though our cur¬ 
rent implementation does not provide a way to pan within or 
scale a canvas, one can use ^ viewBox ^ in order to render the 
output in “full screen” mode. 

The second interesting aspect is the square, which is 
rotated using the SVG ^matrix^ command within the 
^transform^ attribute. Again, even though we currently 
provide no special support for these features, the zone we 
display for the square (though not rotated to match) can still 
be used to directly manipulate it. 

Unlike the Sketch-n-Sketch logo, the high-level re¬ 
lationships between the shapes in the Elm logo are not cap¬ 
tured by the definition in SVG format, nor in its direct trans¬ 
lation to little. As a result, directly manipulating any one 
of the pieces does not affect the others, therefore breaking 
the intended abstraction of the logo. 

Sketch-n-Sketch Logo. To implement our 
logo, which pays homage to the Elm logo, 
we use the abstraction facilities of a program¬ 
ming language to declare relationships be¬ 
tween the shapes. The definition is parame¬ 
terized by a position (xO, yO) for the top-left 
corner, a width w and height h, and a delta parameter that 
determines the size of the gap between the three shapes: 

(let [xO yO w h delta] [50 50 200 200 10] ...) 

The rest of the definition (not shown) computes the three 
polygons in terms of these parameters. It is, thus, simple 
to change any of these values and re-run the program to 
generate an updated logo. 

Better yet is the ability to manipulate the parameters 
directly through the canvas in live mode. Eor example, say 
that we want to stretch the logo, that is, by changing the w 
and h parameters. If we click and drag bottom-right corner 

^ github.com/evancz/elm-svg/blob/1.0.2/examples/Logo.elm 


{i.e. a Point zone) of the bottom triangle in live mode, the 
height of the logo is adjusted but not the width; instead, the 
x-position of the logo is. In other words, the location set 
assigned to this particular zone happens tobc {xq, h} instead 
of /i} as we might have liked. 

We can proceed in a couple of ways. One option is to 
edit the code to freeze the xO, yO, and delta values, thereby 
directing Sketch-n-Sketch towards assigning the desired 
location set and trigger to this POINT zone. With this change, 
directly manipulating this corner point allows us to stretch 
the logo in either direction. 

Another option is to create a transparent rectangle in 
the background with dimensions w and h. The BotRight- 
CORNER zone of this box will, predictably, be assigned the 
location set {w, h}, thus providing direct manipulation con¬ 
trol over the desired attributes of the logo. This second op¬ 
tion, creating an explicit “group box,” is a design pattern that 
is often useful for mixing programmatic and direct manip¬ 
ulation (Le. prodirect manipulation) in the current version 
of Sketch-n-Sketch. In future work, it may be useful to 
provide some built-in support for grouping shapes. 

Chicago Flag. Like with the Sketch-n-Sketch logo, 
we define a transparent group box (visible when display¬ 
ing zones, as in the screenshot) to give direct manipulation 
control over the width and height of the flag. Unlike that 
example, however, there is no way to produce the same 
exact result by manipulating only one of the polygons. 
If the user changes, say, the bottom 
stripe by moving the mouse cursor 
a given distance, the overall dimen¬ 
sions of the fiag will change, but not 
by the amount the cursor has moved. 

As a result, the relationship between 
stretching one of the stripes and the overall fiag is not a 
smooth, intuitive one. Using a group box, however, provides 
the simple and expected behavior. 

Chicago Botanic Garden 

The symmetric design of this logo uses 
curves, defined with Bezier path com¬ 
mands. By programming in little, 
we can define the coordinates and con¬ 
trol points such that they are refiected across a vertical axis 
running down the middle of the logo. Then, in live mode, 
direct manipulation of any position or control point (the 
“fioating” Point zones in the screenshot) in either half is 
immediately refiected in the other half. 

Active Trans The logo of the Active Transporta¬ 

tion Alliance contains two paths, each of which has a single 
curved edge and some number of straight edges. In our cur¬ 
rent implementation, Sketch-n-Sketch does not provide 

^ WWW.chicagobotanic.org 
WWW . activetrans. org 























a GUI-based way to create shapes or add extra points to ex¬ 
isting shapes. Therefore, these two paths must be generated 
using little code, at least initially. 

Nevertheless, we found that we can 
quickly and easily begin implementing this 
logo as follows. First, we implement a 
makePath function that stitches together a 
path based on a list of points and a single 
Bezier control point. Next, we define two 
intially-empty lists, grayPts and greenPts, 
that will store the points of each path. Then, 
we use makePath to construct two paths out of these lists. 

(let makePath (A(color pts [xCtrl yCtrl]) ...) 
(let [grayPts greenPts] [ [] [] ] 

(let [pi p2] [(makePath ...) (makePath ...)] 

...))) 


Ferris Wheel We designed a ferris wheel that consists of 
some number of equal-length spokes emanating from a cen¬ 
tral hub, each of which has a passenger car at its end. Fur¬ 
thermore, we wanted the ability to rotate the wheel while 
keeping the passenger cars vertical, in order to accurately 
portray the physical characteristics of a ferris wheel in mo¬ 
tion. It is hard to imagine how one could develop these re¬ 
lationships in a modular way using tools like Illustrator or 
PowerPoint. 

In Sketch-n-Sketch, we combine programmatic, di¬ 
rect manipulation, and indirect manipulation (via user- 
defined sliders) to develop our design in a way that is highly- 
reusable and easy to edit. First, we write a function 

(def ferrisWheel 

(A(numSpokes spokeLen rotAngle 

sizeCar radiusCenter cx cy) ...)) 



Now the task is to define the list of points for each path. We 
would like to do this visually by directly manipulating points 
into the desired positions, but we need some points to begin 
with. As is, grayPts and greenPts are empty, so there are 
no shapes to render. 

One option is to use a text editor to populate the list with 
dummy points, but this could be tedious for a large num¬ 
ber of points, especially because they should be reasonably 
spaced out so that they can be manipulated in the visual ed¬ 
itor. Instead, we wrote a little function to generate such 
a list of points and evaluated it using the Elm REPL (read- 
eval-print loop). We then copied this list into our program, 
rendered it, and proceeded to directly manipulate the points. 
Our helper function essentially created a “ball of clay” that 
we massaged into the desired shapes. In future work, the 
visual editor might provide support for generating complex 
shapes using templates such as this one. 

Once we settled on the desired shapes of our paths, we 
returned to the program to introduce structure that relates 
the topmost points of the top shape (corresponding to the 
city skyline). As a result, dragging any one of these points 
up or down in live mode affects all of the others. So, if the 
skyline grows taller (which has been known to happen in 
Chicago), we can easily adapt the logo to match. 

Lastly, we include a button in our development and use 
it to toggle between a “positive” version, where the shapes 
are colored and the background is white, and a “negative” 
version, where the shapes are white and the background is 
colored. These two versions of the logo are easy to develop 
in tandem using Sketch-n-Sketch. 


User-Defined Widget s, [Figure 7 shows the definition of 
slider described in |§ 6[ The ghosts function adds the 
^ HIDDEN ^ attribute to a list of shapes so that they can hidden 
by toggling a button. 


(def ghosts 

(map (Ashape (consAttr shape [^HIDDEN^ ^O))) 


that, given several parameters, draws the desired circles, 
lines, and rectangles. The function is straightforward to 
write, making use of a Prelude function 

(def nPointsOnCircle (A(n rot cx cy r) ...)) 

that generates a list of n points evenly spaced around a circle 
of r radius centered at (cx, cy). A drawing that results from 
ferrisWheel is shown in the bottom of |Figure~^ 

We can directly manipulate several parameters of the 
ferris wheel: we can adjust the location (cx,cy) of the 
wheel by dragging the INTERIOR zone of the central hub; 
we can adjust radiusCenter to change the size of the cen¬ 
tral hub by manipulating its Edge zone; and we can ad¬ 
just the width sizeCar of all passenger cars by manipulat¬ 
ing any one of their Edge zones. While this workflow in 
Sketch-n-Sketch is already unique and quite useful, it 
would be nice to also have a way to adjust numSpokes and 
rotAngle in the visual editor. However, no zones are con¬ 
nected to these parameters. 

Therefore, we add sliders to control the parameters 
numSpokes, rotAngle, and spokeLen from the GUI editor. 

(let [num si] (intSlider ... 5^^) 

(let [len s2] (intSlider ... 80 ^^) 

(let [rot s3] (numSlider ... 0^^) 

(let wheel (ferrisWheel num len rot ...) 

(let sliders 
(let show true 

(if show (concat [si s2 s3]) [])) 


(svg (append sliders wheel))))))) 

The resulting canvas is shown in Eigure 8[ With this setup, 
we can easily tweak any of the parameters to ferrisWheel 
in live mode without having to modify the program. If we 
wanted to change something about the ferris wheel abstrac¬ 
tion, of course, we could easily switch to programmatic ma¬ 
nipulation as needed. 










; slider : Bool -> Int -> Int -> Int -> Num -> Num -> Str -> Num 
; -> [Num (List Svg)] 

(def slider (ACroundlnt xO xl y minVal maxVal caption srcVal) 
(let preVal (clamp minVal maxVal srcVal) 

(let targetVal (if roundint (round preVal) preVal) 

(let shapes 
(let ball 

(let [xDiff valDiff] [(- xl xO) (- maxVal minVal)] 

(let xBall (+ xO (* xDiff (/ (- srcVal minVal) valDiff))) 
(if (= preVal srcVal) (circle ^black^ xBall y 10!) 

(circle ^red^ xBall y 10!)))) 

[ (line ^black^ 3! xO y xl y) 

(text (+ xl 10) (+ y 5) (+ caption (toString targetVal))) 
(circle ^black^ xO y 4!) 

(circle ^black^ xl y 4!) 
ball ]) 

[targetVal (ghosts shapes)]))))) 

(def [numSlider intSlider] [(slider false) (slider true)]) 


Figure 7. Horizontal Slider in little. 
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Figure 8. Ferris Wheel Example. 
Generated using a combination of 
programmatic, direct, and user- 
defined indirect manipulation. 


To wrap up, we note how easy it is to export our ferris 
wheel design once we have finished modifying it. We can 
set the show parameter to false in order to hide the slid¬ 
ers from the output. From there, we use the export facility 
in Sketch-n-Sketch to generate the raw SVG for our de¬ 
sign, which we can copy and paste into other SVG editors. 


Procedural vs. Relational Constructions. As mentioned 
^ a detailed comparison of programming graphic de¬ 


signs in these two styles may be an interesting avenue for 
future work. Here, we merely note one relational specifica¬ 
tion that can be easily rephrased procedurally, namely, that 
of an equilateral triangle. We can use the nStar function 
defined earlier to derive an equilateral triangle: 


(def tri (A(c x y sideLen rot) 

(let lent (* sideLen (/ 2! 3!)) 

(let len2 (* sideLen (/ 1! 3!)) 

(nStar c ^none^ 0 3! lent len2 rot x y))))) 


E. User Study 

In this paper, we presented several ways of mitigating am¬ 
biguities to enable live synchronization: freezing constants, 
using heuristics for automatic disambiguation, and using 
sliders. We performed a user study to evaluate the rela¬ 
tive strengths and weaknesses of these techniques. We also 
sought to find out whether the combination of programmatic 
and direct manipulation is a compelling idea. 

Hypotheses. Our aim was to test the following hypotheses: 

1. Simple heuristics for automatic disambiguation are some¬ 
times preferable to using sliders. 

2. Direct manipulation (either with sliders or heuristics) is 
usually preferable to making all edits programmatically. 

3. Adding variables and other programmatic constructs to 
full-featured direct manipulation tools is desired. 

E.l Procedures 

Sketch-n-Sketch is not a full-featured direct manipula¬ 
tion tool {e.g. Illustrator), nor is it a polished language im¬ 
plementation with rich libraries and tools {e.g. Processing). 
As a result, asking users to learn to use Sketch-n-Sketch 
(i) would be time-consuming, (ii) would not allow apples- 
to-apples comparisons with the aforementioned tools, and 
(iii) would not be likely to isolate the interesting questions 
at the boundary between programmatic and direct manipula¬ 
tion. Instead, we designed a series of videos showing expert 
Sketch-n-Sketch users (the authors) working with the 
tool. We asked participants to watch the videos and then an¬ 
swer survey questions based on their observations. For refer¬ 
ence, the survey questions as administered and a link to our 
videos can be found in Supplementary Appendices na. 










Video 0: Background (3 minutes). We described the rel¬ 
ative strengths and weaknesses of programming and direct 
manipulation for creating graphic designs. We then asked 
background questions about programming and graphic de¬ 
sign experience. 

Video 1: Intro to Shetch-n-Sketch (9 minutes). We intro¬ 
duced little, basic SVG features, unambiguous direct ma¬ 
nipulation updates, and freezing constants. 

Video 2: Examples (2 minutes). We showed two examples 
where variables relate different attributes and unambiguous 
direct manipulation updates preserve these relationships. 

Video 3: Heuristics (6 minutes). We described the heuris¬ 
tics for automatic disambiguation. 

Video 4: Sliders (3 minutes). We demonstrated sliders as 
a way to control otherwise hard-to-manipulate parameters. 

Video 5: Side-by-Side Comparisons (22 minutes). This 
section was designed to answer the question, “Are direct ma¬ 
nipulation heuristics better or worse than simply providing a 
slider for every constant in the program?” 

We demonstrated a series of three tasks (Ferris Wheel, 
Keyboard, and Tessellation), each starting with an initial de¬ 
sign (the “Before” column in [Figure 9| ) with the goal of edit¬ 
ing it to realize a target design (the “After” column). We per¬ 
formed each task twice: (A) using only sliders when needing 
to break ambiguities (heuristics were disabled) and (B) rely¬ 
ing on the heuristics and freezing constants to break ambi¬ 
guities (sliders were not allowed). In both interaction modes, 
unambiguous direct manipulation updates were allowed and 
several programmatic edits were required. 

After showing a task performed both ways, we asked 
participants to rate the relative effectiveness of interac¬ 
tion modes (A) and (B). We also asked participants to rate 
each compared to a third mode: (C) using only program¬ 
matic edits (no direct manipulation updates or sliders). Our 
videos did not explicitly demonstrate mode (C). To con¬ 
clude the study, our survey asked about final impressions of 
Sketch-n-Sketch and prodirect manipulation. 

Participants, We sought users with programming experi¬ 
ence, because the current version of Sketch-n-Sketch 
requires it. We advertised our study to undergraduate. Mas¬ 
ters, and PhD students in the Computer Science Department 
at the home institution of the authors. We held three separate, 
in-person sessions where we showed the videos and admin¬ 
istered anonymous surveys. A total of 25 students attended 
the sessions and completed surveys. Each person was paid 
$20 for their participation, except one person who refused 
payment. The study was reviewed and approved by the In¬ 
stitutional Review Board (IRB) at our home institution. 

E.2 Results 

Our participants had significant programming background, 
with 64% reporting at least 3 years of experience. We also 



Figure 9. The goal of each task (Ferris Wheel, Keyboard, 
and Tessellation) was to convert the “Before” design into 
the “After” design. For each task, users evaluated three 
ways of dealing with ambiguities during the editing process: 
(A) Sliders; (B) Heuristics; (C) Programmatic Manipulation 
Only. The “Histograms” show the results. 


found that participants, on average, generate 18% of their 
graphic design work programmatically. 

The side-by-side comparisons for the Ferris Wheel, Key¬ 
board, and Tessellation tasks comprised the primary evalua¬ 
tive aspect of our study. For each pair of interaction modes 
(Ml) and (M2), we provided a five-option, balanced rating 
scale which we interpret as a number in the range [—2, 2], 
where —2 and —1 represent strong and weak preference, re¬ 
spectively, for (Ml) and 1 and 2 represent weak and strong 
preference, respectively, for (M2). The “Histograms” col¬ 
umn of [Figure"^ shows the survey results. Each edge be¬ 
tween modes (Ml) and (M2) of the triangle displays a his¬ 
togram of the relative preferences between (Ml) and (M2). 
We calculated the means along with 95% bootstrap-t confi¬ 
dence intervals,which are displayed with red lines along 
the edges of the triangle. 

Hypothesis 1, This table summarizes the mean prefer¬ 
ence ratings, along with 95% confidence intervals, between 
sliders (A) and heuristics (B) for each of the three tasks. 
Neither sliders nor heuristics 
were preferred for the Fer¬ 
ris Wheel task (F), heuristics 
were weakly preferred over 
sliders for the Keyboard task 
(K), and neither was pre¬ 
ferred for the Tessellation task (T). These data suggest that 
even simple heuristics can provide an advantage over sliders. 



(A) vs. (B) 

F 

-0.52 (-0.92, 0.01) 

K 

0.76 ( 0.26,1.18) 

T 

0.20 (-0.20, 0.64) 


































Therefore, we conclude that developing smarter heuristics in 
future work may provide an even more desirable workflow. 

Hypothesis 2. Compared to manual code edits (C), both 
sliders (A) and heuristics (B) were, on average, preferred on 
every task. The following summarizes the mean preference 
ratings and 95% confidence intervals: 



(C) vs. (A) 

(C) vs. (B) 

F 

1.12 (0.59, 1.47) 

0.80 (0.25, 1.23) 

K 

0.92 (0.59, 1.21) 

1.24 (0.73, 1.57) 

T 

0.76 (0.34, 1.10) 

1.00 (0.53, 1.32) 


We deliberately did not present tasks that could be best ac¬ 
complished through manual code edits, so this weak pref¬ 
erence cannot be generalized to all graphics manipulation 
tasks. However, the data does suggest that the ways in which 
Sketch-n-Sketch provides interactive, direct manipula¬ 
tion are sometimes preferable to purely programmatic ma¬ 
nipulation. 

Hypothesis 3. Participants reported that, on average, 50% 
of designs they have created using direct manipulation tools 
would have benefited from programmatic manipulation. Our 
participants had significant programming background, so 
this percentage is likely to be higher than if participants had 
been drawn from a wider variety of users. Nevertheless, this 
finding supports the need for combining programmatic and 
direct manipulation at least among expert programmers. 

Threats to Validity, Because we showed each task twice, 
insights from the first observation may have affected the 
second. To mitigate learning effects, we performed similar 
edits and narration at the same pace in the second version, 
even when they were redundant when viewed after the first. 

Choices in program representation have a significant im¬ 
pact on what kinds of direct manipulation may or may not be 
possible. In an attempt not to bias our initial programs (used 
to generate the “Before” designs) towards either mode (A) 
or (B), we wrote near-final drafts of the programs before de¬ 
ciding exactly what manipulation tasks to perform. We also 
chose the specific tasks by taking inspiration from outside 
references, such as photos. 

Finally, the participants made judgments about the rela¬ 
tive merits of sliders and heuristics based on observations, 
not by using the tool directly. We plan to run “hands-on” 
experiments in the future, which will first require achieving 
more feature-parity with existing tools. 

Notes 

^ A. Canty and B. Ripley, “Package Boot,” https: //cran. r-pro j ect. 
org/web/packages/boot/boot.pdf 

^A. C. Davison and D. Kuonen, “An Introduction to the Bootstrap with 
Applications in R,” 2002 

^A. C. Davison and D. V. Hinkley, “Bootstrap Methods and Their Ap¬ 
plication,” 1997 


F. User Study Survey 

This section provides the survey instrument that we adminis¬ 
tered (reformatted for this paper). For multiple-choice ques¬ 
tions, we display the number of participants (out of 25) that 
chose each option in parentheses. Our videos can be found 
on our project Web page. 

Background Questions 

How often do you use graphic design applications? 

(Examples: Adobe Illustrator, Microsoft PowerPoint, GIMP) 

• Less than once a year (0) 

• A few times a year (9) 

• A few times a month (11) 

• A few times a week (5) 

• Every day or almost every day (0) 

How many years of programming experience do you have? 

• Less than 1 (3) 

• 1-2 ( 6 ) 

• 3-5 (8) 

• 6-10 ( 8 ) 

• 11 - 20 ( 0 ) 

• More than 20 (0) 

What percent of your graphic design work is programmatically 
generated? (Place a mark on the line.) 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 

Side By Side Comparison: Ferris Wheel 

Interaction A: Sliders and unambiguous direct manipulation. 
Interaction B: Direct manipulation with heuristics and freezing. 

Which interaction worked better for the Perris Wheel task? 

• Interaction A worked much better (3) 

• Interaction A worked a little better (14) 

• They are about the same (2) 

• Interaction B worked a little better (5) 

• Interaction B worked much better (1) 

The next two questions ask you to consider how each interaction 
would compare to just making all edits in the code manually. 

Por the Perris Wheel task, how good was interaction A? 

• Manual code edits only would have worked much better (0) 

• Manual code edits only would have worked a little better (3) 

• About the same as manual code edits (1) 

• Interaction A worked a little better (11) 

• Interaction A worked much better (10) 

Por the Perris Wheel task, how good was interaction B? 

• Manual code edits only would have worked much better (1) 

• Manual code edits only would have worked a little better (3) 

• About the same as manual code edits (4) 

• Interaction B worked a little better (9) 

• Interaction B worked much better (8) 






Side By Side Comparison: Keyboard 


Additional Questions 


Interaction A: Sliders and unambiguous direct manipulation. 
Interaction B: Direct manipulation with heuristics and freezing. 

Which interaction worked better for the Keyboard task? 

• Interaction A worked much better (0) 

• Interaction A worked a little better (5) 

• They are about the same (3) 

• Interaction B worked a little better (10) 

• Interaction B worked much better (7) 

The next two questions ask you to consider how each interaction 
would compare to just making all edits in the code manually. 

For the Keyboard task, how good was interaction A? 

• Manual code edits only would have worked much better (0) 

• Manual code edits only would have worked a little better (1) 

• About the same as manual code edits (5) 

• Interaction A worked a little better (14) 

• Interaction A worked much better (5) 

For the Keyboard task, how good was interaction B? 

• Manual code edits only would have worked much better (0) 

• Manual code edits only would have worked a little better (2) 

• About the same as manual code edits (2) 

• Interaction B worked a little better (9) 

• Interaction B worked much better (12) 

Side By Side Comparison: Tessellation 

Interaction A: Sliders and unambiguous direct manipulation. 
Interaction B: Direct manipulation with heuristics and freezing. 

Which interaction worked better for the Tessellation task? 

• Interaction A worked much better (0) 

• Interaction A worked a little better (7) 

• They are about the same (9) 

• Interaction B worked a little better (6) 

• Interaction B worked much better (3) 

The next two questions ask you to consider how each interaction 
would compare to just making all edits in the code manually. 

For the Tessellation task, how good was interaction A? 

• Manual code edits only would have worked much better (1) 

• Manual code edits only would have worked a little better (0) 

• About the same as manual code edits (8) 

• Interaction A worked a little better (11) 

• Interaction A worked much better (5) 

For the Tessellation task, how good was interaction B? 

• Manual code edits only would have worked much better (1) 

• Manual code edits only would have worked a little better (0) 

• About the same as manual code edits (4) 

• Interaction B worked a little better (13) 

• Interaction B worked much better (7) 


Thinking of all the graphics you have created with direct manip¬ 
ulation tools, what percentage of those graphics would have been 
easier to create if your direct manipulation tool also allowed pro¬ 
grammatic manipulation? (Place a mark on the line.) 

I-^^^^^^^^^-1 

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 

Do you plan to try using Sketch-n-Sketch to create graphics? 

• Certainly not (0) 

• Probably not (2) 

• Maybe (8) 

• Likely (12) 

• Certainly (3) 

What improvements or new features would make Sketch-n-Sketch 
better? 

Are there any other comments about Sketch-n-Sketch or the idea 
of prodirect manipulation that you would like to share? 

Today you saw a demonstration of prodirect manipulation for 
graphics editing. Are there other applications where you would 
like to see prodirect manipulation? 



G. Measurements 


The following measurements, summarized and discussed 
[§5) were collected on vO. 4.2 of our implementation. 






Number of zones with 

Example Name 

Shape Count 

Zone Count 

0 

, 1, > 1 loclist choices 




0 

1 

> 1 (avg) 

Wave Boxes 

12 

108 

0 

36 

72 (2.67) 

Wave Boxes Grid 

78 

702 

18 

246 

438 (10.71) 

Logo 

4 

30 

12 

9 

9 (3.22) 

Botanic Garden Logo 

4 

25 

1 

23 

1 (2.00) 

Active Trans Logo 

9 

40 

8 

32 

0 

Sailboat 

36 

132 

15 

75 

42 (4.07) 

Chicago Flag 

7 

127 

4 

73 

50 (5.84) 

Sliders 

24 

34 

30 

4 

0 

Buttons 

6 

7 

6 

1 

0 

Widgets 

24 

33 

29 

4 

0 

xySlider 

13 

19 

18 

1 

0 

Tile Pattern 

49 

123 

104 

19 

0 

Color Picker 

25 

36 

32 

4 

0 

Ferris Wheel 

13 

80 

11 

49 

20 (2.25) 

Ferris Task Before 

13 

80 

0 

49 

31 (3.45) 

Ferris Task After 

19 

125 

0 

76 

49 (3.47) 

Ferris Wheel Slideshow 

14 

18 

16 

2 

0 

Survey Results 

207 

752 

188 

564 

0 

Hilbert Curve Animation 

16 

21 

16 

5 

0 

Bar Graph 

48 

60 

25 

30 

5 (3.00) 

Pie Chart 

41 

105 

91 

5 

9 (6.00) 

Solar System 

23 

41 

8 

25 

8 (7.00) 

Clique 

37 

105 

0 

105 

0 

Eye Icon 

4 

21 

7 

12 

2 (2.00) 

Wikimedia Logo 

6 

20 

10 

10 

0 

Haskell.org Logo 

4 

22 

2 

8 

12 (2.50) 

Cover Logo 

16 

159 

9 

150 

0 

POP-PL Logo 

8 

36 

22 

12 

2 (2.00) 

Lillicon P 

8 

30 

7 

21 

2 (2.00) 

Lillicon P, v2 

3 

15 

2 

13 

0 

Keyboard 

40 

360 

0 

61 

299 (6.17) 

Keyboard Task Before 

28 

252 

0 

1 

251 (5.82) 

Keyboard Task After 

33 

297 

0 

7 

290 (5.81) 

Tessellation Task Before 

1364 

4992 

0 

1536 

3456 (2.56) 

Tessellation Task After 

980 

2496 

0 

384 

2112(2.45) 

Floral Logo 1 

20 

140 

0 

0 

140 (7.86) 

Floral Logo 2 

18 

126 

0 

0 

126 (8.52) 

Spiral Spiral-Graph 

140 

420 

0 

280 

140 (12.00) 

Rounded Rect 

12 

28 

8 

14 

6 (2.33) 

Thaw/Freeze 

1 

9 

5 

4 

0 

3 Boxes 

3 

27 

0 

19 

8 (2.00) 

N Boxes Sli 

3 

27 

0 

19 

8 (2.00) 

N Boxes 

10 

49 

5 

27 

17 (3.65) 

Elm Logo 

7 

53 

0 

53 

0 

Logo 2 

4 

46 

15 

22 

9 (2.22) 

Logo Sizes 

12 

138 

61 

52 

25 (2.08) 

Rings 

5 

10 

0 

6 

4 (2.50) 

Polygons 

5 

73 

0 

39 

34 (5.00) 

Stars 

5 

105 

7 

48 

50 (2.88) 

Triangles 

2 

26 

0 

0 

26 (2.46) 

US-13Plag 

22 

354 

14 

203 

137 (3.90) 

US-50 Elag 

59 

181 

14 

110 

57 (3.75) 

Trench Sudan Elag 

12 

59 

11 

31 

17 (3.76) 

Erank Lloyd Wright 

31 

119 

42 

9 

68 (5.06) 

Bezier Curves 

22 

42 

11 

19 

12 (7.50) 

Eractal Tree 

27 

60 

18 

4 

38 (13.74) 

Stick Eigures 

54 

193 

53 

56 

84 (3.92) 

Matrix Transformations 

5 

20 

0 

0 

20 (3.20) 

Misc Shapes 

6 

30 

0 

30 

0 

Interface Buttons 

21 

70 

14 

32 

24 (2.21) 

Paths 1 

3 

8 

0 

8 

0 

Paths 2 

9 

36 

0 

36 

0 

Paths 3 

3 

13 

0 

13 

0 

Paths 4 

1 

6 

0 

6 

0 

Paths 5 

4 

12 

0 

12 

0 

Sample Rotations 

2 

15 

0 

14 

1 (9.00) 

Grid Tile 

24 

79 

22 

14 

43 (2.98) 

Zones 

4 

29 

0 

24 

5 (5.00) 


Totals 

3772 

14106 

991 

4856 

8259 (3.83) 




Example Name 

# Output Locs 

Uufrozeu 

Uuassigued 

Assigued (avg times) (avg rate) 

Wave Boxes 

7 

6 

0 

6 (40.0) (67%) 

Wave Boxes Grid 

29 

24 

1 

23 (64.5) (34%) 

Logo 

6 

6 

1 

5 (7.6) (36%) 

Botanic Garden Logo 

22 

19 

7 

12 (3.2) (98%) 

Active Trans Logo 

71 

44 

1 

43 (1.5) (100%) 

Sailboat 

52 

38 

22 

16 (14.3) (85%) 

Chicago Flag 

19 

8 

1 

7 (24.4) (23%) 

Sliders 

32 

12 

8 

4 (1.0) (100%) 

Buttons 

9 

3 

2 

1 (1.0) (100%) 

Widgets 

18 

4 

0 

4 (1.0) (100%) 

xySlider 

18 

7 

5 

2 (1.0) (100%) 

Tile Pattern 

49 

27 

0 

27 (1.1) (100%) 

Color Picker 

27 

4 

0 

4 (1.0) (100%) 

Ferris Wheel 

11 

9 

2 

7(11.0) (53%) 

Ferris Task Before 

11 

11 

2 

9 (9.6) (42%) 

Ferris Task After 

11 

11 

2 

9 (14.9) (42%) 

Ferris Wheel Slideshow 

22 

12 

10 

2 (1.0) (100%) 

Survey Results 

94 

64 

63 

1 (564.0) (100%) 

Hilbert Curve Animation 

44 

34 

27 

7 (1.0) (100%) 

Bar Graph 

40 

5 

0 

5 (9.0) (46%) 

Pie Chart 

46 

13 

7 

6 (2.3) (24%) 

Solar System 

38 

16 

0 

16 (2.8) (74%) 

Clique 

14 

14 

1 

13 (20.5) (100%) 

Eye Icon 

40 

22 

15 

7 (2.0) (90%) 

Wikimedia Logo 

19 

8 

4 

4 (3.0) (85%) 

Haskell.org Logo 

58 

23 

16 

7 (3.1) (59%) 

Cover Logo 

15 

2 

0 

2 (75.0) (100%) 

POP-PL Logo 

25 

13 

0 

13 (1.8) (96%) 

Lillicon P 

18 

15 

1 

14 (3.3) (98%) 

Lillicon P, v2 

19 

17 

4 

13 (2.0) (100%) 

Keyboard 

45 

45 

11 

34 (18.1) (34%) 

Keyboard Task Before 

44 

44 

2 

42 (13.3) (43%) 

Keyboard Task After 

46 

46 

2 

44 (14.5) (38%) 

Tessellation Task Before 

89 

77 

72 

5 (1510.2) (57%) 

Tessellation Task After 

87 

76 

71 

5 (768.0) (60%) 

Floral Logo 1 

33 

32 

1 

31 (4.7) (16%) 

Floral Logo 2 

32 

29 

1 

28 (4.7) (16%) 

Spiral Spiral-Graph 

14 

14 

0 

14 (30.9) (18%) 

Rounded Rect 

16 

11 

4 

7 (5.9) (85%) 

Thaw/Freeze 

4 

1 

0 

1 (4.0) (100%) 

3 Boxes 

5 

5 

0 

5 (12.0) (83%) 

N Boxes Sli 

5 

5 

0 

5 (12.0) (83%) 

N Boxes 

22 

17 

6 

11 (7.8) (76%) 

Elm Logo 

43 

43 

1 

42 (4.1) (100%) 

Logo 2 

9 

7 

1 

6 (8.2) (66%) 

Logo Sizes 

15 

11 

1 

10 (11.7) (48%) 

Rings 

6 

6 

1 

5 (3.0) (77%) 

Polygons 

29 

26 

2 

24 (4.1) (28%) 

Stars 

8 

6 

1 

5 (22.6) (25%) 

Triangles 

14 

9 

1 

8 (3.8) (29%) 

US-13 Flag 

27 

7 

1 

6 (76.5) (26%) 

US-50 Flag 

26 

5 

0 

5 (54.8) (66%) 

French Sudan Flag 

21 

4 

0 

4 (24.0) (55%) 

Frank Lloyd Wright 

40 

31 

6 

25 (5.8) (60%) 

Bezier Curves 

23 

12 

1 

11 (5.2) (57%) 

Fractal Tree 

37 

22 

4 

18 (3.5) (18%) 

Stick Figures 

85 

68 

13 

55 (6.3) (81%) 

Matrix Transformations 

57 

7 

1 

6 (3.3) (43%) 

Misc Shapes 

28 

28 

3 

25 (3.3) (100%) 

Interface Buttons 

35 

18 

12 

6(13.3) (58%) 

Paths 1 

19 

19 

3 

16 (1.0) (100%) 

Paths 2 

72 

72 

0 

72 (1.0) (100%) 

Paths 3 

26 

26 

0 

26 (1.0) (100%) 

Paths 4 

24 

24 

12 

12 (1.0) (100%) 

Paths 5 

48 

48 

24 

24 (1.0) (100%) 

Sample Rotations 

12 

12 

2 

10 (2.8) (97%) 

Grid Tile 

14 

8 

2 

6 (19.0) (62%) 

Zones 

31 

28 

1 

27 (2.0) (87%) 

Totals 

2075 

1440 

465 

975 (21.1) (69%) 



# Equations 

# (example, loc, tr) Equations 
Mean Trace Size (tree nodes) 
Max Solve Time 

Median Solve Time 
Mean Solve Time 

# Traces in Solve A fragment 

Solved 
+ = 1 
+ = 100 
Not Solved 
+ = 1 
+ = 100 

# Traces in Solve B fragment 

Solved 

+ = 1 
+ = 100 

Not Solved 

+ = 1 
+ = 100 

# Traces in either fragment 

Solved 

+ = 1 
+ = 100 

Not Solved 

+ = 1 
+ = 100 

# Traces in no fragment 


28222 

4574 

141.30 

0.014 

< 0.001 (Calculated: 0.0000) 

< 0.001 (Calculated: 0.0001) 

778 

778 

778 

0 

0 

3655 

3461 

3023 

194 

632 

3656 

3462 

3024 

194 

632 

918 



Example Name 

Parse 

Eval 

Unparse 

valToInde... 

prepareLi... 

“Run Code” 

Wave Boxes 

FF: 0.016 

FF: 0.001 

FF: 0.001 

FF: 0.000 

FF: 0.027 

FF: 0.050 

LOG: 7 

Gh: 0.020 

Gh: 0.003 

Gh: 0.001 

Gh: 0.000 

Gh: 0.061 

Gh: 0.090 

Wave Boxes Grid 

FF: 0.100 

FF: 0.012 

FF: 0.004 

FF: 0.001 

FF: 3.372 

FF: 3.510 

LOG: 42 

Gh: 0.131 

Gh: 0.025 

Gh: 0.004 

Gh: 0.002 

Gh: 6.174 

Ch: 6.365 

Logo 

FF: 0.027 

FF: 0.007 

FF: 0.001 

FF: 0.000 

FF: 0.006 

FF: 0.052 

LOG: 18 

Gh: 0.035 

Gh: 0.026 

Gh: 0.002 

Gh: 0.000 

Gh: 0.012 

Ch: 0.106 

Botanic Garden Logo 

FF: 0.060 

FF: 0.001 

FF: 0.001 

FF: 0.002 

FF: 0.005 

FF: 0.074 

LOG: 28 

Gh: 0.072 

Gh: 0.002 

Gh: 0.003 

Gh: 0.001 

Gh: 0.005 

Ch: 0.089 

Active Trans Logo 

FF: 0.067 

FF: 0.004 

FF: 0.004 

FF: 0.001 

FF: 0.010 

FF: 0.095 

LOG: 33 

Gh: 0.089 

Gh: 0.008 

Gh: 0.002 

Gh: 0.002 

Gh: 0.019 

Ch: 0.131 

Sailboat 

FF: 0.089 

FF: 0.009 

FF: 0.004 

FF: 0.004 

FF: 0.030 

FF: 0.151 

LOG: 51 

Gh: 0.115 

Gh: 0.012 

Gh: 0.004 

Gh: 0.007 

Gh: 0.053 

Ch: 0.210 

Ghicago Flag 

FF: 0.032 

FF: 0.011 

FF: 0.002 

FF: 0.000 

FF: 0.046 

FF: 0.104 

LOG: 16 

Gh: 0.040 

Gh: 0.025 

Gh: 0.001 

Gh: 0.000 

Gh: 0.087 

Ch: 0.178 

Sliders 

FF: 0.083 

FF: 0.001 

FF: 0.003 

FF: 0.000 

FF: 0.002 

FF: 0.093 

LOG: 38 

Gh: 0.105 

Gh: 0.004 

Gh: 0.003 

Gh: 0.001 

Gh: 0.004 

Ch: 0.122 

Buttons 

FF: 0.033 

FF: 0.001 

FF: 0.003 

FF: 0.000 

FF: 0.002 

FF: 0.043 

LOG: 16 

Gh: 0.039 

Gh: 0.000 

Gh: 0.002 

Gh: 0.000 

Gh: 0.002 

Ch: 0.046 

Widgets 

FF: 0.015 

FF: 0.003 

FF: 0.000 

FF: 0.000 

FF: 0.002 

FF: 0.025 

LOG: 5 

Gh: 0.018 

Gh: 0.008 

Gh: 0.001 

Gh: 0.000 

Gh: 0.005 

Ch: 0.045 

xySlider 

FF: 0.078 

FF: 0.002 

FF: 0.003 

FF: 0.000 

FF: 0.003 

FF: 0.092 

LOG: 33 

Gh: 0.098 

Gh: 0.002 

Gh: 0.003 

Gh: 0.000 

Gh: 0.003 

Ch: 0.112 

Tile Pattern 

FF: 0.121 

FF: 0.033 

FF: 0.004 

FF: 0.000 

FF: 0.004 

FF: 0.202 

LOG: 74 

Gh: 0.154 

Gh: 0.050 

Gh: 0.005 

Gh: 0.001 

Gh: 0.009 

Ch: 0.272 

Golor Picker 

FF: 0.025 

FF: 0.003 

FF: 0.001 

FF: 0.000 

FF: 0.003 

FF: 0.039 

LOG: 8 

Gh: 0.028 

Gh: 0.006 

Gh: 0.001 

Gh: 0.001 

Gh: 0.003 

Ch: 0.046 

Ferris Wheel 

FF: 0.038 

FF: 0.004 

FF: 0.004 

FF: 0.000 

FF: 0.010 

FF: 0.066 

LOG: 12 

Gh: 0.044 

Gh: 0.008 

Gh: 0.002 

Gh: 0.000 

Gh: 0.016 

Ch: 0.083 

Ferris Task Before 

FF: 0.030 

FF: 0.005 

FF: 0.001 

FF: 0.000 

FF: 0.018 

FF: 0.062 

LOG: 16 

Gh: 0.038 

Gh: 0.009 

Gh: 0.001 

Gh: 0.000 

Gh: 0.028 

Ch: 0.090 

Ferris Task After 

FF: 0.031 

FF: 0.008 

FF: 0.002 

FF: 0.000 

FF: 0.023 

FF: 0.073 

LOG: 16 

Gh: 0.037 

Gh: 0.013 

Gh: 0.002 

Gh: 0.001 

Gh: 0.041 

Ch: 0.109 

Ferris Wheel Slideshow 

FF: 0.298 

FF: 0.028 

FF: 0.011 

FF: 0.000 

FF: 0.003 

FF: 0.372 

LOG: 252 

Gh: 0.375 

Gh: 0.063 

Gh: 0.012 

Gh: 0.001 

Gh: 0.004 

Ch: 0.524 

Survey Results 

FF: 0.225 

FF: 0.068 

FF: 0.008 

FF: 0.004 

FF: 0.045 

FF: 0.415 

LOG: 121 

Gh: 0.270 

Gh: 0.157 

Gh: 0.010 

Gh: 0.006 

Gh: 0.082 

Ch: 0.691 

Hilbert Gurve Animation 

FF: 0.409 

FF: 0.006 

FF: 0.010 

FF: 0.000 

FF: 0.006 

FF: 0.443 

LOG: 289 

Gh: 0.503 

Gh: 0.010 

Gh: 0.011 

Gh: 0.000 

Gh: 0.006 

Ch: 0.542 

Bar Graph 

FF: 0.111 

FF: 0.003 

FF: 0.003 

FF: 0.000 

FF: 0.005 

FF: 0.130 

LOG: 43 

Gh: 0.145 

Gh: 0.008 

Gh: 0.005 

Gh: 0.001 

Gh: 0.011 

Ch: 0.182 

Pie Ghart 

FF: 0.089 

FF: 0.004 

FF: 0.003 

FF: 0.001 

FF: 0.007 

FF: 0.112 

LOG: 36 

Gh: 0.112 

Gh: 0.015 

Gh: 0.006 

Gh: 0.002 

Gh: 0.015 

Ch: 0.170 

Solar System 

FF: 0.065 

FF: 0.003 

FF: 0.003 

FF: 0.000 

FF: 0.008 

FF: 0.087 

LOG: 31 

Gh: 0.084 

Gh: 0.006 

Gh: 0.002 

Gh: 0.000 

Gh: 0.014 

Ch: 0.114 

Glique 

FF: 0.027 

FF: 0.006 

FF: 0.001 

FF: 0.000 

FF: 0.023 

FF: 0.066 

LOG: 13 

Gh: 0.032 

Gh: 0.011 

Gh: 0.001 

Gh: 0.001 

Gh: 0.042 

Ch: 0.103 

Eye Icon 

FF: 0.049 

FF: 0.001 

FF: 0.003 

FF: 0.001 

FF: 0.005 

FF: 0.065 

LOG: 37 

Gh: 0.062 

Gh: 0.001 

Gh: 0.002 

Gh: 0.003 

Gh: 0.006 

Ch: 0.079 



Example Name 

Parse 

Eval 

Unparse 

valToInde... 

prepareLi... 

“Run Code” 

Wikimedia Logo 

FF: 0.039 

FF: 0.001 

FF: 0.001 

FF: 0.001 

FF: 0.003 

FF: 0.048 

LOG: 19 

Gh: 0.052 

Gh: 0.001 

Gh: 0.001 

Gh: 0.001 

Gh: 0.003 

Ch: 0.061 

Haskell.org Logo 

FF: 0.093 

FF: 0.001 

FF: 0.002 

FF: 0.001 

FF: 0.003 

FF: 0.103 

LOG: 59 

Gh: 0.119 

Gh: 0.002 

Gh: 0.003 

Gh: 0.001 

Gh: 0.006 

Ch: 0.137 

Gover Logo 

FF: 0.072 

FF: 0.003 

FF: 0.003 

FF: 0.000 

FF: 0.010 

FF: 0.094 

LOG: 39 

Gh: 0.102 

Gh: 0.006 

Gh: 0.005 

Gh: 0.000 

Gh: 0.019 

Ch: 0.144 

POP-PL Logo 

FF: 0.071 

FF: 0.001 

FF: 0.003 

FF: 0.000 

FF: 0.003 

FF: 0.081 

LOG: 57 

Gh: 0.089 

Gh: 0.002 

Gh: 0.003 

Gh: 0.001 

Gh: 0.007 

Ch: 0.108 

Lillicon P 

FF: 0.039 

FF: 0.000 

FF: 0.002 

FF: 0.000 

FF: 0.003 

FF: 0.048 

LOG: 28 

Gh: 0.048 

Gh: 0.002 

Gh: 0.002 

Gh: 0.000 

Gh: 0.007 

Ch: 0.064 

Lillicon P, v2 

FF: 0.026 

FF: 0.000 

FF: 0.001 

FF: 0.000 

FF: 0.002 

FF: 0.033 

LOG: 19 

Gh: 0.031 

Gh: 0.001 

Gh: 0.001 

Gh: 0.000 

Gh: 0.005 

Ch: 0.042 

Keyboard 

FF: 0.110 

FF: 0.006 

FF: 0.004 

FF: 0.000 

FF: 0.873 

FF: 1.005 

LOG: 71 

Gh: 0.140 

Gh: 0.017 

Gh: 0.004 

Gh: 0.001 

Gh: 1.319 

Ch: 1.498 

Keyboard Task Before 

FF: 0.067 

FF: 0.006 

FF: 0.002 

FF: 0.000 

FF: 1.077 

FF: 1.163 

LOG: 48 

Gh: 0.077 

Gh: 0.007 

Gh: 0.002 

Gh: 0.000 

Gh: 1.843 

Ch: 1.939 

Keyboard Task After 

FF: 0.069 

FF: 0.005 

FF: 0.002 

FF: 0.000 

FF: 1.340 

FF: 1.422 

LOG: 51 

Gh: 0.083 

Gh: 0.010 

Gh: 0.003 

Gh: 0.001 

Gh: 2.127 

Ch: 2.235 

Tessellation Task Before 

FF: 0.116 

FF: 0.002 

FF: 0.004 

FF: 0.096 

FF: 0.675 

FF: 0.995 

LOG: 80 

Gh: 0.142 

Gh: 0.003 

Gh: 0.004 

Gh: 0.164 

Gh: 1.319 

Ch: 1.803 

Tessellation Task After 

FF: 0.098 

FF: 0.002 

FF: 0.002 

FF: 0.072 

FF: 0.357 

FF: 0.607 

LOG: 73 

Gh: 0.127 

Gh: 0.004 

Gh: 0.005 

Gh: 0.115 

Gh: 0.671 

Ch: 1.044 

Floral Logo 1 

FF: 0.073 

FF: 0.012 

FF: 0.004 

FF: 0.004 

FF: 0.158 

FF: 0.275 

LOG: 45 

Gh: 0.091 

Gh: 0.028 

Gh: 0.004 

Gh: 0.005 

Gh: 0.379 

Ch: 0.542 

Floral Logo 2 

FF: 0.095 

FF: 0.013 

FF: 0.005 

FF: 0.004 

FF: 0.152 

FF: 0.293 

LOG: 49 

Gh: 0.108 

Gh: 0.025 

Gh: 0.004 

Gh: 0.005 

Gh: 0.345 

Ch: 0.521 

Spiral Spiral-Graph 

FF: 0.034 

FF: 0.030 

FF: 0.002 

FF: 0.002 

FF: 0.257 

FF: 0.351 

LOG: 16 

Gh: 0.041 

Gh: 0.060 

Gh: 0.002 

Gh: 0.003 

Gh: 0.571 

Ch: 0.745 

Rounded Rect 

FF: 0.026 

FF: 0.003 

FF: 0.001 

FF: 0.001 

FF: 0.004 

FF: 0.039 

LOG: 14 

Gh: 0.032 

Gh: 0.004 

Gh: 0.001 

Gh: 0.000 

Gh: 0.005 

Ch: 0.048 

Thaw/Freeze 

FF: 0.015 

FF: 0.000 

FF: 0.001 

FF: 0.000 

FF: 0.003 

FF: 0.021 

LOG: 2 

Gh: 0.011 

Gh: 0.000 

Gh: 0.003 

Gh: 0.000 

Gh: 0.003 

Ch: 0.021 

3 Boxes 

FF: 0.018 

FF: 0.003 

FF: 0.002 

FF: 0.001 

FF: 0.011 

FF: 0.037 

LOG: 7 

Gh: 0.016 

Gh: 0.001 

Gh: 0.001 

Gh: 0.000 

Gh: 0.007 

Ch: 0.028 

N Boxes Sli 

FF: 0.016 

FF: 0.001 

FF: 0.001 

FF: 0.001 

FF: 0.005 

FF: 0.027 

LOG: 7 

Gh: 0.018 

Gh: 0.001 

Gh: 0.001 

Gh: 0.000 

Gh: 0.009 

Ch: 0.033 

N Boxes 

FF: 0.042 

FF: 0.003 

FF: 0.002 

FF: 0.001 

FF: 0.012 

FF: 0.067 

LOG: 24 

Gh: 0.053 

Gh: 0.006 

Gh: 0.002 

Gh: 0.000 

Gh: 0.022 

Ch: 0.091 

Elm Logo 

FF: 0.029 

FF: 0.000 

FF: 0.001 

FF: 0.000 

FF: 0.018 

FF: 0.051 

LOG: 12 

Gh: 0.036 

Gh: 0.002 

Gh: 0.001 

Gh: 0.000 

Gh: 0.034 

Ch: 0.077 

Logo 2 

FF: 0.049 

FF: 0.009 

FF: 0.001 

FF: 0.000 

FF: 0.004 

FF: 0.073 

LOG: 18 

Gh: 0.060 

Gh: 0.026 

Gh: 0.003 

Gh: 0.000 

Gh: 0.010 

Ch: 0.129 

Logo Sizes 

FF: 0.072 

FF: 0.027 

FF: 0.003 

FF: 0.001 

FF: 0.021 

FF: 0.151 

LOG: 33 

Gh: 0.084 

Gh: 0.059 

Gh: 0.002 

Gh: 0.000 

Gh: 0.031 

Ch: 0.244 

Rings 

FF: 0.029 

FF: 0.002 

FF: 0.002 

FF: 0.000 

FF: 0.002 

FF: 0.039 

LOG: 12 

Gh: 0.034 

Gh: 0.004 

Gh: 0.001 

Gh: 0.000 

Gh: 0.002 

Ch: 0.047 

Polygons 

FF: 0.031 

FF: 0.003 

FF: 0.003 

FF: 0.000 

FF: 0.027 

FF: 0.068 

LOG: 15 

Gh: 0.039 

Gh: 0.008 

Gh: 0.001 

Gh: 0.000 

Gh: 0.065 

Ch: 0.126 



Example Name 

Parse 

Eval 

Unparse 

valToInde... 

prepareLi... 

“Run Code” 

Stars 

FF: 0.043 

FF: 0.009 

FF: 0.002 

FF: 0.000 

FF: 0.021 

FF: 0.085 

LOG: 18 

Gh: 0.049 

Gh: 0.023 

Gh: 0.002 

Gh: 0.000 

Gh: 0.040 

Ch: 0.139 

Triangles 

FF: 0.017 

FF: 0.002 

FF: 0.001 

FF: 0.000 

FF: 0.005 

FF: 0.034 

LOG: 14 

Gh: 0.018 

Gh: 0.006 

Gh: 0.000 

Gh: 0.000 

Gh: 0.014 

Ch: 0.046 

US-13 Flag 

FF: 0.048 

FF: 0.017 

FF: 0.003 

FF: 0.000 

FF: 0.107 

FF: 0.198 

LOG: 24 

Gh: 0.060 

Gh: 0.059 

Gh: 0.003 

Gh: 0.001 

Gh: 0.224 

Ch: 0.409 

US-50 Flag 

FF: 0.050 

FF: 0.007 

FF: 0.002 

FF: 0.001 

FF: 0.032 

FF: 0.107 

LOG: 17 

Gh: 0.055 

Gh: 0.016 

Gh: 0.002 

Gh: 0.001 

Gh: 0.053 

Ch: 0.146 

French Sudan Flag 

FF: 0.055 

FF: 0.002 

FF: 0.002 

FF: 0.000 

FF: 0.010 

FF: 0.073 

LOG: 27 

Gh: 0.067 

Gh: 0.003 

Gh: 0.002 

Gh: 0.000 

Gh: 0.016 

Ch: 0.092 

Frank Lloyd Wright 

FF: 0.075 

FF: 0.004 

FF: 0.003 

FF: 0.000 

FF: 0.080 

FF: 0.168 

LOG: 40 

Gh: 0.101 

Gh: 0.012 

Gh: 0.004 

Gh: 0.001 

Gh: 0.157 

Ch: 0.293 

Bezier Gurves 

FF: 0.209 

FF: 0.006 

FF: 0.006 

FF: 0.001 

FF: 0.012 

FF: 0.242 

LOG: 124 

Gh: 0.270 

Gh: 0.012 

Gh: 0.007 

Gh: 0.002 

Gh: 0.021 

Ch: 0.326 

Fractal Tree 

FF: 0.095 

FF: 0.015 

FF: 0.005 

FF: 0.001 

FF: 0.075 

FF: 0.218 

LOG: 38 

Gh: 0.115 

Gh: 0.023 

Gh: 0.004 

Gh: 0.001 

Gh: 0.145 

Ch: 0.310 

Stick Figures 

FF: 0.120 

FF: 0.012 

FF: 0.003 

FF: 0.000 

FF: 0.273 

FF: 0.422 

LOG: 60 

Gh: 0.139 

Gh: 0.025 

Gh: 0.006 

Gh: 0.001 

Gh: 0.490 

Ch: 0.693 

Matrix Transformations 

FF: 0.091 

FF: 0.043 

FF: 0.004 

FF: 0.001 

FF: 0.288 

FF: 0.471 

LOG: 30 

Gh: 0.116 

Gh: 0.126 

Gh: 0.004 

Gh: 0.001 

Gh: 0.730 

Ch: 1.105 

Misc Shapes 

FF: 0.018 

FF: 0.000 

FF: 0.000 

FF: 0.000 

FF: 0.010 

FF: 0.033 

LOG: 8 

Gh: 0.020 

Gh: 0.001 

Gh: 0.001 

Gh: 0.000 

Gh: 0.012 

Ch: 0.036 

Interface Buttons 

FF: 0.225 

FF: 0.007 

FF: 0.011 

FF: 0.003 

FF: 0.010 

FF: 0.267 

LOG: 79 

Gh: 0.276 

Gh: 0.012 

Gh: 0.012 

Gh: 0.000 

Gh: 0.017 

Ch: 0.332 

Paths 1 

FF: 0.012 

FF: 0.000 

FF: 0.000 

FF: 0.002 

FF: 0.003 

FF: 0.021 

LOG: 5 

Gh: 0.013 

Gh: 0.000 

Gh: 0.000 

Gh: 0.001 

Gh: 0.003 

Ch: 0.021 

Paths 2 

FF: 0.027 

FF: 0.000 

FF: 0.001 

FF: 0.001 

FF: 0.012 

FF: 0.043 

LOG: 11 

Gh: 0.034 

Gh: 0.000 

Gh: 0.001 

Gh: 0.002 

Gh: 0.022 

Ch: 0.063 

Paths 3 

FF: 0.012 

FF: 0.000 

FF: 0.001 

FF: 0.001 

FF: 0.004 

FF: 0.020 

LOG: 5 

Gh: 0.014 

Gh: 0.000 

Gh: 0.001 

Gh: 0.000 

Gh: 0.005 

Ch: 0.023 

Paths 4 

FF: 0.013 

FF: 0.001 

FF: 0.001 

FF: 0.000 

FF: 0.003 

FF: 0.020 

LOG: 11 

Gh: 0.015 

Gh: 0.000 

Gh: 0.000 

Gh: 0.001 

Gh: 0.004 

Ch: 0.024 

Paths 5 

FF: 0.018 

FF: 0.001 

FF: 0.001 

FF: 0.001 

FF: 0.003 

FF: 0.028 

LOG: 10 

Gh: 0.025 

Gh: 0.000 

Gh: 0.000 

Gh: 0.001 

Gh: 0.005 

Ch: 0.036 

Sample Rotations 

FF: 0.034 

FF: 0.004 

FF: 0.003 

FF: 0.000 

FF: 0.004 

FF: 0.050 

LOG: 16 

Gh: 0.044 

Gh: 0.003 

Gh: 0.001 

Gh: 0.000 

Gh: 0.005 

Ch: 0.058 

Grid Tile 

FF: 0.050 

FF: 0.010 

FF: 0.005 

FF: 0.000 

FF: 0.020 

FF: 0.097 

LOG: 23 

Gh: 0.053 

Gh: 0.010 

Gh: 0.001 

Gh: 0.000 

Gh: 0.023 

Ch: 0.102 

Zones 

FF: 0.043 

FF: 0.005 

FF: 0.004 

FF: 0.001 

FF: 0.021 

FF: 0.081 

LOG: 14 

Gh: 0.039 

Gh: 0.003 

Gh: 0.001 

Gh: 0.001 

Gh: 0.015 

Ch: 0.063 


FF: 0.069 

FF: 0.007 

FF: 0.003 

FF: 0.003 

FF: 0.142 

FF: 0.238 


Gh: 0.085 

Gh: 0.016 

Gh: 0.003 

Gh: 0.005 

Gh: 0.257 

Ch: 0.390 

Summary 

Min 0.009 

Min 0.000 

Min 0.000 

Min 0.000 

Min 0.001 

Min 0.017 


Med 0.053 

Med 0.005 

Med 0.002 

Med 0.000 

Med 0.013 

Med 0.098 


Mean 0.077 

Mean 0.012 

Mean 0.003 

Mean 0.004 

Mean 0.200 

Mean 0.314 


Max 0.520 

Max 0.165 

Max 0.019 

Max 0.180 

Max 6.789 

Max 6.980 



